+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20201015/B1e --output-run DEEP/20201015/B1e/science#step1/20240313T183244Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 4.5099 seconds; current memory usage: 0.180 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.192 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0097 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0108 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/pipeline_orig.qgraph DEEP/20201015/B1e/science#step1/20240313T183244Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 6.1797 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 16.9118 seconds; current memory usage: 0.384 Gibyte, delta: 0.204 Gibyte, peak delta: 0.192 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.384 Gibyte (main), 0.371 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0303 seconds; current memory usage: 0.384 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.384 Gibyte (main), 0.371 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 1116 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20201015_B1e_science#step1_20240313T183244Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.2209 seconds; current memory usage: 0.384 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.384 Gibyte (main), 0.371 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 1118 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.5422 seconds; current memory usage: 0.386 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.386 Gibyte (main), 0.371 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 7193722e-d57e-4006-8252-0b5b9c521b7c parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-13 18:33:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-13 18:33:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 18:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-13 18:33:35 proc_lsst.multi:170 [INFO] [multi] job_id 98521 proc_lsst.multi INFO: [multi] job_id 98521 2024-03-13 18:33:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:35 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 98521 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 98521 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 98521 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 95bf04ef6f0edfa9344ff7fce60a72c2 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247915580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/6dda49f4-adac-4661-9b27-3540837c24d8_isr_945757_54.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/6dda49f4-adac-4661-9b27-3540837c24d8_isr_945757_54.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1 parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 0eee71cabc609a56c552a7e7510f0a0e parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247915e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d0180f42-548b-4c95-a348-681b7ecc1b21_isr_945868_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d0180f42-548b-4c95-a348-681b7ecc1b21_isr_945868_11.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3 parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 1cccbf907f3c5a55d1c67b89b3fc51fd parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479160c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e1850ac7-9092-445b-90f1-f95a6842ccf2_isr_945869_5.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e1850ac7-9092-445b-90f1-f95a6842ccf2_isr_945869_5.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7 parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 1e94ee661bd7df06c2e36d9c819945e9 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479162a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/6628ee47-f59d-458b-a784-5a1d50149b94_isr_945869_29.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/6628ee47-f59d-458b-a784-5a1d50149b94_isr_945869_29.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9 parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash b5ebd4b35b97993aca7f3bda0b657d32 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479167a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/a51d3912-9d6f-4279-bfad-0172fa492418_isr_945868_4.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/a51d3912-9d6f-4279-bfad-0172fa492418_isr_945868_4.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13 parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 64c5605ccaa598c2bb3277cc3f853566 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247916de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/ea174d62-ed86-46ca-8299-a78c0aa30985_isr_945869_62.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/ea174d62-ed86-46ca-8299-a78c0aa30985_isr_945869_62.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15 parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash abd370f213879030511aa5946e4297dc parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247916f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/855a30c6-301b-48ad-8111-e92db41b3a87_isr_945755_24.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/855a30c6-301b-48ad-8111-e92db41b3a87_isr_945755_24.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19 parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 37c0c5d53b6e00fbb43922ee41b19164 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479171a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/f2cf5d78-f158-48de-9297-fc5ba1a558fb_isr_945868_35.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/f2cf5d78-f158-48de-9297-fc5ba1a558fb_isr_945868_35.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 2fa4dd38b1613d740d26384afbb7f4ac parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247917d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/568fb02f-20cc-4514-a446-edccc967c773_isr_945868_29.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/568fb02f-20cc-4514-a446-edccc967c773_isr_945868_29.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25 parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 9d16a44d35f7f703b711b7bf334e0c03 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247917a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/54b0bec5-ceef-4fd4-b384-3b11305de974_isr_945755_23.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/54b0bec5-ceef-4fd4-b384-3b11305de974_isr_945755_23.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27 parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 3d43bdbba5df6c059905e7ff7e8f7721 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247917740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2764af42-aeeb-4859-b050-5e57f7a0055a_isr_945867_7.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2764af42-aeeb-4859-b050-5e57f7a0055a_isr_945867_7.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31 parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 5e619aa3ed93ce47d39a23b66442e01d parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247917420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/0b541f54-f937-4e50-850b-be875a489dad_isr_945869_53.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/0b541f54-f937-4e50-850b-be875a489dad_isr_945869_53.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33 parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash eac62ec2a975e2b5b5e37a29056348fc parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/334a821e-6b0a-4170-86e8-234f929e04a5_isr_945755_37.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/334a821e-6b0a-4170-86e8-234f929e04a5_isr_945755_37.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37 parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash e2d4b67c16171d4d6461c5c281b710f3 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/2feafb51-ca72-4661-a6b5-eaae709d1522_isr_945869_4.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/2feafb51-ca72-4661-a6b5-eaae709d1522_isr_945869_4.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39 parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 1e49d86749baa4122bcb594813e38b84 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8a25dc69-dae0-4847-bdaf-cd12a4894de0_isr_945868_23.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8a25dc69-dae0-4847-bdaf-cd12a4894de0_isr_945868_23.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43 parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 6f29130236ee59c9906ccfccfe49d2c0 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/35a9409f-0acf-4fa9-8116-c5cf586602ed_isr_945869_20.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/35a9409f-0acf-4fa9-8116-c5cf586602ed_isr_945869_20.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45 parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash ff30ac4bff00a19dc132f99747c19a4b parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d45f4bcf-91ee-42a3-a921-e4170d81cb00_isr_945755_2.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d45f4bcf-91ee-42a3-a921-e4170d81cb00_isr_945755_2.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49 parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash b1b16ed221e283ed091f59be94d79d15 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/90f9e654-017d-451c-967a-6fdcffff674c_isr_945867_36.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/90f9e654-017d-451c-967a-6fdcffff674c_isr_945867_36.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51 parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash c2bf5683cb795afa32313250ffb63f26 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/bb3c2a6c-ce7b-4362-8a5f-c3d16a576d12_isr_945757_22.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/bb3c2a6c-ce7b-4362-8a5f-c3d16a576d12_isr_945757_22.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55 parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 8fc4b7c61129ef4659726908e4cf8780 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1601abcc-244c-4b4c-873e-d630e0f88df5_isr_945867_37.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1601abcc-244c-4b4c-873e-d630e0f88df5_isr_945867_37.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57 parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash de232b03c7e1027d4596195bd1aa934f parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/b7eead58-2e80-4c9d-9fd0-5d0f95f9d347_isr_945756_11.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/b7eead58-2e80-4c9d-9fd0-5d0f95f9d347_isr_945756_11.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61 parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash e5bc421a8f7e49c81198bcfe508d44f5 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d7c2c130-d4e5-4228-b908-c14d76da36f1_isr_945757_7.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d7c2c130-d4e5-4228-b908-c14d76da36f1_isr_945757_7.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63 parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 760240fb5b7b270952c95a42c9492521 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a0099220-29ce-48a4-8f70-7b702553d321_isr_945757_34.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a0099220-29ce-48a4-8f70-7b702553d321_isr_945757_34.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 3401458397aded6038207cee4610dc60 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/e00001b2-0d0e-435b-a8ad-a587c225bd5a_isr_945867_48.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/e00001b2-0d0e-435b-a8ad-a587c225bd5a_isr_945867_48.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69 parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 8df3a9398552b8d607f7790a8e19d640 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8130c8fc-9433-43c5-a63e-7204cab5522e_isr_945868_19.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8130c8fc-9433-43c5-a63e-7204cab5522e_isr_945868_19.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73 parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 6abc3b2715967ad5040f6c5d78ba5eac parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/1b793783-5e3f-4f2e-b705-d5e26158e70a_isr_945757_55.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/1b793783-5e3f-4f2e-b705-d5e26158e70a_isr_945757_55.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75 parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 951df5cde2f0aac7b4072bcff2fbbab1 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/3878e9c7-89b4-4eda-af3d-ae36970b01a9_isr_945868_24.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/3878e9c7-89b4-4eda-af3d-ae36970b01a9_isr_945868_24.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79 parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash a66b5fc7793e565724a70fece02d57a2 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e2588901-0392-4c0c-9af8-eedfeb1afff7_isr_945869_41.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e2588901-0392-4c0c-9af8-eedfeb1afff7_isr_945869_41.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81 parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 91ecf30da07a9e05d5bed5a1330575a8 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/7ce1a1b3-1f93-4865-8022-b6981ba6bd4c_isr_945869_8.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/7ce1a1b3-1f93-4865-8022-b6981ba6bd4c_isr_945869_8.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85 parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash e0ed62dea488147ddeeaf114ae102afa parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c2a995fe-d799-4b3f-8063-2c7f677b067b_isr_945868_15.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c2a995fe-d799-4b3f-8063-2c7f677b067b_isr_945868_15.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87 parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 8fb8491ea13e330d0661bb7da4a2f346 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/520a0f09-31a5-4740-a159-d7ee25d25e91_isr_945757_51.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/520a0f09-31a5-4740-a159-d7ee25d25e91_isr_945757_51.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91 parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash c61d5f9104d4d953780a6e8fb49a204a parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/9500330a-d7a6-43c1-b380-6334db5b4a79_isr_945756_26.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/9500330a-d7a6-43c1-b380-6334db5b4a79_isr_945756_26.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93 parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash e009cf1b511b2a7da45fc65b8cf59090 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d58748a3-37f1-41ee-b826-001c157c0105_isr_945755_55.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d58748a3-37f1-41ee-b826-001c157c0105_isr_945755_55.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97 parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 78d1d9e6d5fb1e31c224dc731d416dcb parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/70fabf7d-91a9-4ac0-9cf8-dc83acca4296_isr_945756_55.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/70fabf7d-91a9-4ac0-9cf8-dc83acca4296_isr_945756_55.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99 parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 57d1a09c1ec7327088896bd167bb6b9e parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/6a82f918-cdf5-4dce-968f-988166418a3b_isr_945757_40.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/6a82f918-cdf5-4dce-968f-988166418a3b_isr_945757_40.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103 parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash bfb2c902f0ef8928942c9150fb09358a parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/716615b0-56be-4640-b9b1-f7143953cea9_isr_945868_49.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/716615b0-56be-4640-b9b1-f7143953cea9_isr_945868_49.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105 parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 23b52474f0947c5b8757c8941fdd6fe3 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/84cb85c2-c72f-4e2a-b874-502221962f5f_isr_945755_42.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/84cb85c2-c72f-4e2a-b874-502221962f5f_isr_945755_42.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109 parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 1792032cfb4806661963d6709db55be3 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/27d7efcc-4123-41f9-8c71-2dcdb092d103_isr_945756_54.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/27d7efcc-4123-41f9-8c71-2dcdb092d103_isr_945756_54.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111 parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 12fc8abc15cccab5ceb9fc745663706e parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8df7866a-3a82-4662-b9ac-f170cc16ada2_isr_945755_59.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8df7866a-3a82-4662-b9ac-f170cc16ada2_isr_945755_59.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115 parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash f76fb3d6f11cf2ec6aec8b9c823e36b2 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/1a229376-2bea-43e6-b91b-2aaae18f39a5_isr_945757_45.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/1a229376-2bea-43e6-b91b-2aaae18f39a5_isr_945757_45.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117 parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 3627e23f063da9cf8aba46e47f004daf parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/127df8ec-3426-4d59-8c94-c0eb60c6b618_isr_945867_38.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/127df8ec-3426-4d59-8c94-c0eb60c6b618_isr_945867_38.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121 parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash f5114f08b3b4ddbcff4b45e315ad62ff parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f25cbaf3-3cbf-41b0-921f-651950182b2c_isr_945757_50.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f25cbaf3-3cbf-41b0-921f-651950182b2c_isr_945757_50.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123 parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash e4b6283de55b3791e1a84356f1f59430 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2d731711-e69c-4338-bc73-c5bc7f608f1d_isr_945867_27.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2d731711-e69c-4338-bc73-c5bc7f608f1d_isr_945867_27.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127 parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash c7663c1820e80d88d4573cfc67426d8e parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/aea6b6a9-7968-41e3-a509-f864a74b99a9_isr_945867_21.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/aea6b6a9-7968-41e3-a509-f864a74b99a9_isr_945867_21.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129 parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 791e723a82fb04de22f56b9a1f31dd60 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/12fb4da3-6bec-4479-86b7-0db1aa950526_isr_945867_45.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/12fb4da3-6bec-4479-86b7-0db1aa950526_isr_945867_45.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133 parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 42f45cebc9d26ad38c35d188061e3b9b parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/52d6697c-a763-4a0e-bf21-6e3edcb0606e_isr_945757_62.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/52d6697c-a763-4a0e-bf21-6e3edcb0606e_isr_945757_62.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135 parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 130ea5583e3c8e7e007afad4ab0d0707 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/37907bcf-16b3-412d-bc75-599938697f8d_isr_945869_48.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/37907bcf-16b3-412d-bc75-599938697f8d_isr_945869_48.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139 parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 411ac38f93924c17b937e0a7e2eec688 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8b1bc25c-0f71-4535-accb-29ca454aef38_isr_945867_2.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8b1bc25c-0f71-4535-accb-29ca454aef38_isr_945867_2.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141 parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash df9b45bd95a3164487f7ccdb926be953 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/345b8026-6065-420e-af1d-d0527e11b890_isr_945867_30.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/345b8026-6065-420e-af1d-d0527e11b890_isr_945867_30.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145 parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 8b1f619092dc27fdf84876605f6af6b7 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b01fa0cd-e0cc-456a-a4ad-847a8ef3ad93_isr_945757_49.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b01fa0cd-e0cc-456a-a4ad-847a8ef3ad93_isr_945757_49.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147 parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash c57e86bd6c070ea2c8a383a2683052cc parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/9ee6bfd5-475d-43e4-94fb-f35baff641cb_isr_945756_62.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/9ee6bfd5-475d-43e4-94fb-f35baff641cb_isr_945756_62.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151 parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 2a808d864dd2220aa9037cb562866a35 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b6810ff9-46bb-457a-9eb2-99af8d4dad17_isr_945757_8.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b6810ff9-46bb-457a-9eb2-99af8d4dad17_isr_945757_8.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153 parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 402016ac0f8a6f37023440b72d0008f3 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/03432321-7b36-4944-a3f4-306cb2729efa_isr_945756_61.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/03432321-7b36-4944-a3f4-306cb2729efa_isr_945756_61.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157 parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 6c1bc45ac2c20e930abfa3237f9cf9be parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/bb2ad845-20de-45d4-b9fd-dff5d1a24a4e_isr_945757_14.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/bb2ad845-20de-45d4-b9fd-dff5d1a24a4e_isr_945757_14.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash db7cfd1e9c3ee7204d7d7e699aefe44c parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/e8251338-0caa-4265-a6b8-1bf59dab0dbf_isr_945756_40.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/e8251338-0caa-4265-a6b8-1bf59dab0dbf_isr_945756_40.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163 parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash cbf3b018691645fad93d051d109a1760 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8206b796-9922-4405-ba9f-d10c4134f0d9_isr_945869_25.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8206b796-9922-4405-ba9f-d10c4134f0d9_isr_945869_25.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165 parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 0ac8ea4f946e838bf70d521b5641ce42 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9fa1bfec-01d2-4291-b7de-64d123f9fc77_isr_945868_43.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9fa1bfec-01d2-4291-b7de-64d123f9fc77_isr_945868_43.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169 parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 2fa174848abb0c5418fd5b144e37aa8c parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/dab39a29-ef94-44be-b4a5-f207e0f6bb03_isr_945869_9.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/dab39a29-ef94-44be-b4a5-f207e0f6bb03_isr_945869_9.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171 parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 50e31a0dd53520c13d623c74d0de8951 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/2d0e8ccb-bf49-4012-83f7-bc62eda4f091_isr_945869_51.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/2d0e8ccb-bf49-4012-83f7-bc62eda4f091_isr_945869_51.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175 parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 5f0e9c031edfa6f275501cee3d25a46c parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/aa55f5ae-6c78-4e59-a789-4bbcdf22463a_isr_945755_7.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/aa55f5ae-6c78-4e59-a789-4bbcdf22463a_isr_945755_7.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177 parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash d2767198565a002be6a6f607d37b3f80 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/17b72214-7865-40f2-af1c-60b18cfdf995_isr_945869_27.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/17b72214-7865-40f2-af1c-60b18cfdf995_isr_945869_27.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181 parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 80fe0d8a3d2b9571149b583ea7ff8804 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f493f01b-dd9e-4e89-9122-d722f79dc4af_isr_945757_59.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f493f01b-dd9e-4e89-9122-d722f79dc4af_isr_945757_59.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183 parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 9b06fb01b0e15ce9077f0ceb80df6122 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/6062b832-5b72-4a16-b900-3505ab21d07e_isr_945755_52.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/6062b832-5b72-4a16-b900-3505ab21d07e_isr_945755_52.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187 parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 37c2557dee120c3750dbdda997e9d89c parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/1acce9a5-3095-412c-991c-fe9041b4f7e1_isr_945757_39.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/1acce9a5-3095-412c-991c-fe9041b4f7e1_isr_945757_39.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189 parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 7833f1a46d85408d01255367a383dd96 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/3c1f00f3-63ba-4d43-a289-51c7fef60722_isr_945868_37.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/3c1f00f3-63ba-4d43-a289-51c7fef60722_isr_945868_37.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193 parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 40cfdaa316a60e00a4c27a7aa77ebfb5 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8b7a13a0-3ecd-44f8-a182-ae3d97e7116a_isr_945868_27.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8b7a13a0-3ecd-44f8-a182-ae3d97e7116a_isr_945868_27.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195 parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 4f986582d6ca132719948b649bb3a2ff parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/9ff6e045-1e3c-4989-8018-214b4e9ebcde_isr_945757_32.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/9ff6e045-1e3c-4989-8018-214b4e9ebcde_isr_945757_32.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199 parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 8af4c97e30ba0c2057f9dd437195d6ca parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/20d67d8b-cb1a-49ce-a16f-736841bcf2db_isr_945757_56.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/20d67d8b-cb1a-49ce-a16f-736841bcf2db_isr_945757_56.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201 parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 3fc7fb31e4e28e7bb49838eab0282d65 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/cf0d5b35-7fb0-47bf-bcf6-d50f70851b30_isr_945755_15.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/cf0d5b35-7fb0-47bf-bcf6-d50f70851b30_isr_945755_15.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 074170d09172d25870ba9e0c204105f6 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/0f84601d-49ad-4909-a667-e5cae5c30914_isr_945869_56.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/0f84601d-49ad-4909-a667-e5cae5c30914_isr_945869_56.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207 parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash a62829fd628964e13ecf695739c1ffdc parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2aaf167b-6527-462b-9a67-2b5c5b70bfff_isr_945756_60.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2aaf167b-6527-462b-9a67-2b5c5b70bfff_isr_945756_60.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211 parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 2741ca1733a8955a695fbc932c1ee282 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/37e02578-614f-4e56-9640-f1e69d7bdeb4_isr_945869_15.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/37e02578-614f-4e56-9640-f1e69d7bdeb4_isr_945869_15.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213 parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 5b5dca17209335a1565baeb0668b32a9 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9f09fa07-9bd5-4ed6-afae-81c94ce72a9f_isr_945868_58.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9f09fa07-9bd5-4ed6-afae-81c94ce72a9f_isr_945868_58.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217 parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash b2a1544a90f643e075015fe6893a06bf parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247994220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/4c9173f9-91e3-46e3-8281-6b7608cc6251_isr_945869_10.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/4c9173f9-91e3-46e3-8281-6b7608cc6251_isr_945869_10.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219 parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash f08c42737a48cd5e384e3c3286bdbe63 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247994540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/23210f94-6771-4a7a-a569-7d05930f35ac_isr_945756_36.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/23210f94-6771-4a7a-a569-7d05930f35ac_isr_945756_36.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223 parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 341d57cf688c1c08e190a9f4ccac2fff parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247994860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2f9b4c8d-833a-4127-9ae5-7a98653d084d_isr_945867_40.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2f9b4c8d-833a-4127-9ae5-7a98653d084d_isr_945867_40.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225 parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 3ddd6aaea8a518b3f7d973a84adc5321 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247994b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b0ac4ad1-cb33-4808-a5ef-0330b498f927_isr_945757_41.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b0ac4ad1-cb33-4808-a5ef-0330b498f927_isr_945757_41.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229 parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 65cdb16a72e8c60efbba8311fc39d907 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247994ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/b9c84918-88e4-409f-83f3-96c953605470_isr_945867_15.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/b9c84918-88e4-409f-83f3-96c953605470_isr_945867_15.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231 parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash e030f2747e0b39823269643fd533b799 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479951c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b2230e79-109c-49b2-bdfa-1c9e367f93db_isr_945757_26.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b2230e79-109c-49b2-bdfa-1c9e367f93db_isr_945757_26.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235 parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 1602d816f6f587a406c6fae2f6794bc6 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479954e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/ad6d2f8b-3f1d-4106-9ed2-8bb34985e13d_isr_945867_46.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/ad6d2f8b-3f1d-4106-9ed2-8bb34985e13d_isr_945867_46.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237 parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 7f2021958bfb8e7bc006ddd7231ca933 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247995800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/37eb98c5-af48-41a3-8859-799bb9fbdc7f_isr_945757_17.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/37eb98c5-af48-41a3-8859-799bb9fbdc7f_isr_945757_17.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241 parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 983b1fab660916c4b3965032fc4fc782 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247995b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/275ad293-2145-40a3-a83e-ee2a0a9e9485_isr_945867_42.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/275ad293-2145-40a3-a83e-ee2a0a9e9485_isr_945867_42.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243 parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 1cc33adfbb116ff5d628adf24bf8358b parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247995e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/d8a6d391-a68c-4e45-8f31-992773b48f4d_isr_945867_56.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/d8a6d391-a68c-4e45-8f31-992773b48f4d_isr_945867_56.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247 parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash f20152eb276b0dceb39fd868396c11ff parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247996160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/435c45eb-a359-4abe-8643-4c17158853bf_isr_945757_35.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/435c45eb-a359-4abe-8643-4c17158853bf_isr_945757_35.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249 parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250 parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 5c76856dad52c164cf0955caa1ed6727 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247996480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/90f30185-a306-4462-9d85-60bad92c968e_isr_945868_61.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/90f30185-a306-4462-9d85-60bad92c968e_isr_945868_61.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252 parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253 parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 9ca4cf6c9c9ea99873132f2a275f8a1a parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479967a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/25285027-166e-4f9b-978b-3b74e024c4e6_isr_945755_62.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/25285027-166e-4f9b-978b-3b74e024c4e6_isr_945755_62.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255 parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256 parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 86d17962a77e318c7aa377d8ffa8ae65 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247996ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ecccf29d-a108-44a0-b624-618879e7fc8d_isr_945868_56.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ecccf29d-a108-44a0-b624-618879e7fc8d_isr_945868_56.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258 parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259 parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash c840ffa375daac2c136a635d1879ee8c parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247996de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/db837592-ceca-4c75-9171-79f3da080deb_isr_945867_17.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/db837592-ceca-4c75-9171-79f3da080deb_isr_945867_17.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261 parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262 parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 54776e958db437b7f7cf4681f6a81319 parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f3422e49-6e2d-4220-9cd9-4279b536f08d_isr_945869_46.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f3422e49-6e2d-4220-9cd9-4279b536f08d_isr_945869_46.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264 parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265 parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 68699e1b6471980b2c5aeedddf8dd709 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/de16d5e0-be22-4403-82c2-b46a9da83182_isr_945756_19.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/de16d5e0-be22-4403-82c2-b46a9da83182_isr_945756_19.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267 parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268 parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash b77142b43c7ceed187db6108c80e26fd parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/2bea1c82-4da0-4cfa-825e-794273f4f9ff_isr_945757_20.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/2bea1c82-4da0-4cfa-825e-794273f4f9ff_isr_945757_20.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270 parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271 parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 020e9f00768988927f5cff16c8047640 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4a04a615-9c12-488b-bfe9-2fbe44b1b570_isr_945867_5.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4a04a615-9c12-488b-bfe9-2fbe44b1b570_isr_945867_5.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273 parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 3989e486e6f7d698ad855cd96193b89f parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3e89c763-eb4d-438b-8ae2-cde9d8ec057d_isr_945755_32.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3e89c763-eb4d-438b-8ae2-cde9d8ec057d_isr_945755_32.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276 parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277 parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 2a1ad88fbc89b82f6bb1e817ef3e51f6 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463900e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/31716c3e-36bf-4de2-85a5-508d49e24aca_isr_945755_41.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/31716c3e-36bf-4de2-85a5-508d49e24aca_isr_945755_41.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279 parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280 parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 42db85f1984e3958e54029c3ccfc379b parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246390400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d4bd6a9d-01b4-4f37-a3e5-6135aa0edff9_isr_945757_5.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d4bd6a9d-01b4-4f37-a3e5-6135aa0edff9_isr_945757_5.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282 parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283 parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 96d324796d286a32f720bc92d732948d parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246390720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/c5140c1e-14fa-43be-810b-fdd3d7c73bd6_isr_945869_54.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/c5140c1e-14fa-43be-810b-fdd3d7c73bd6_isr_945869_54.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285 parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286 parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 7cdd1f5802d975c7e3221808ba58bf74 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246390a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/fd243099-165c-430f-aff4-87d1e3d88941_isr_945757_57.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/fd243099-165c-430f-aff4-87d1e3d88941_isr_945757_57.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288 parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289 parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash ea01b32649ee54911fe3e22c8319bb1c parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246390d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/befb6d0e-1ecf-4ea4-90af-be0c806d84aa_isr_945868_50.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/befb6d0e-1ecf-4ea4-90af-be0c806d84aa_isr_945868_50.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291 parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292 parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 705e329b8d4bfc70ef3863c7a1381c7f parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246391080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/77ad538e-3061-4a5a-a0a9-42ff4f1d1f12_isr_945869_36.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/77ad538e-3061-4a5a-a0a9-42ff4f1d1f12_isr_945869_36.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294 parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295 parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 3ebf206d76a85dd46c6af0a365237864 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463913a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/63749640-5019-4166-b298-aa9d7b1fd6b6_isr_945869_26.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/63749640-5019-4166-b298-aa9d7b1fd6b6_isr_945869_26.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298 parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 7b26f623e554e746b31ede39ccfec9a4 parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463916c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/ca46f251-0cae-4928-9ce6-e2f88aeb8092_isr_945755_39.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/ca46f251-0cae-4928-9ce6-e2f88aeb8092_isr_945755_39.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300 parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301 parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash b2d844416bd0d38878791ea713d8dcb1 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463919e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/f9986f8f-57f5-4ab9-9fb9-3aad74cc4c06_isr_945756_35.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/f9986f8f-57f5-4ab9-9fb9-3aad74cc4c06_isr_945756_35.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303 parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304 parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 2cd168ba5457f63de5dae6745a290668 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246391d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/932e36b6-de5b-44c1-8fd4-470b2da7e3e9_isr_945756_38.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/932e36b6-de5b-44c1-8fd4-470b2da7e3e9_isr_945756_38.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306 parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307 parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash da11c63b9f3260167ff0d7c2a61697f9 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246392020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/4627d01a-47ea-4b70-a44d-74618a2ab684_isr_945868_53.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/4627d01a-47ea-4b70-a44d-74618a2ab684_isr_945868_53.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309 parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310 parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 5f020fff9c68793b2ca6755f79fbccac parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246392340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/dcc2269f-b3f7-4b53-9dc5-7998b419274a_isr_945868_12.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/dcc2269f-b3f7-4b53-9dc5-7998b419274a_isr_945868_12.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312 parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313 parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 27b2d2c5c350844076ab072aacf666d9 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246392660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/22f4b161-dbc2-4945-8268-b1fb7c6207eb_isr_945755_1.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/22f4b161-dbc2-4945-8268-b1fb7c6207eb_isr_945755_1.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315 parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316 parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 09f9203defd8f436654ef6dd49ab2bed parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246392980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/6a8eb84e-bfb2-434b-9aac-4a079169b7ab_isr_945868_17.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/6a8eb84e-bfb2-434b-9aac-4a079169b7ab_isr_945868_17.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318 parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319 parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash e9cc6305db387a23b28e5db0a127b1c1 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246392ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/812993e5-60a3-46e1-9da8-dca34e21d2fb_isr_945755_3.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/812993e5-60a3-46e1-9da8-dca34e21d2fb_isr_945755_3.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321 parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322 parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 1edf7f1c1dedbc22a5e21176c6d26d62 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246392fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/1e29b5b3-a49d-48e7-9a84-067d5b0c02e8_isr_945755_36.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/1e29b5b3-a49d-48e7-9a84-067d5b0c02e8_isr_945755_36.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324 parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325 parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 255cc49da3ec553fb7344b07461cf182 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463932e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/187dfb6a-7589-40c4-9e50-b34c7b0f6404_isr_945757_43.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/187dfb6a-7589-40c4-9e50-b34c7b0f6404_isr_945757_43.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327 parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328 parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 640d4e235f340ecda40de4e1a761e0bf parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/616cd863-39be-4738-9e2d-6e6615c99856_isr_945755_34.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/616cd863-39be-4738-9e2d-6e6615c99856_isr_945755_34.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330 parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331 parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 396e6a5f83542d6f54b8e4a38415ded1 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/df8d1bcf-494f-4d16-871b-0f9f86619948_isr_945757_48.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/df8d1bcf-494f-4d16-871b-0f9f86619948_isr_945757_48.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333 parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334 parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash d6b544821ebbfd38952db9995ff56cf2 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/7f7bee1f-14c7-42f8-b5a9-37e18a788a51_isr_945868_7.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/7f7bee1f-14c7-42f8-b5a9-37e18a788a51_isr_945868_7.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336 parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337 parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 6fc1bdd2380685797830c4842b3a2960 parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/41fa5242-4fac-48a6-8b97-aae13d5654f4_isr_945867_31.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/41fa5242-4fac-48a6-8b97-aae13d5654f4_isr_945867_31.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339 parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340 parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash c1f10890ad7be0a55ff62029f4ab0145 parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/5c77a25f-dee2-4034-b206-7c389526f07e_isr_945756_9.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/5c77a25f-dee2-4034-b206-7c389526f07e_isr_945756_9.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342 parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash ff826a1b7d5075bf9b78ca92942d7478 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/1d7b01e8-3e9c-4bfb-8578-c5f7cc8d6261_isr_945869_60.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/1d7b01e8-3e9c-4bfb-8578-c5f7cc8d6261_isr_945869_60.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345 parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346 parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 2c2ac4b5627d2af83487738303d90ad2 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/6b125acb-fe7e-4241-9ddb-202f06ee9820_isr_945867_9.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/6b125acb-fe7e-4241-9ddb-202f06ee9820_isr_945867_9.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348 parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349 parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 32557df8f0cf5ec70dfa3b069000e418 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1ceaf3ba-aac3-4502-97d9-a47af4188a2f_isr_945867_57.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1ceaf3ba-aac3-4502-97d9-a47af4188a2f_isr_945867_57.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351 parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352 parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash a43217d925e0aa653e0639bb7d45fd10 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a39e3c74-1523-4301-9d8d-7e24adaa9a18_isr_945757_1.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a39e3c74-1523-4301-9d8d-7e24adaa9a18_isr_945757_1.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354 parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355 parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 06ed5230df4dcc41f1ddbf4f604dac75 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/96c83b62-ac0b-461f-9606-9474cf6ed92e_isr_945868_60.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/96c83b62-ac0b-461f-9606-9474cf6ed92e_isr_945868_60.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357 parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358 parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 0af449bcb07a3f7dd64d2259255e98d1 parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/7ba95dfe-5790-4aed-a6aa-61e6c4622ac8_isr_945757_42.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/7ba95dfe-5790-4aed-a6aa-61e6c4622ac8_isr_945757_42.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360 parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361 parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 013f70142a6257ba30a9728416433f5a parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/fc348cb2-b398-4eaf-9588-35a4febf349e_isr_945757_15.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/fc348cb2-b398-4eaf-9588-35a4febf349e_isr_945757_15.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363 parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364 parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash eb3a65b1f6c6fccb27f014e2c3a23a3b parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3eb21e85-30ab-4586-a8d6-4f4abea41e1a_isr_945755_20.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3eb21e85-30ab-4586-a8d6-4f4abea41e1a_isr_945755_20.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367 parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash fe41cd355ec554a2fcd1a5db54daf2af parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/02464a48-7f01-48d8-b791-fac5800e03a4_isr_945867_29.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/02464a48-7f01-48d8-b791-fac5800e03a4_isr_945867_29.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369 parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370 parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash b313be17ab47027c752ec5466344831c parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 18:33:36 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/2efdbcfe-617a-4f43-a54e-fd049d581042_isr_945755_57.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/2efdbcfe-617a-4f43-a54e-fd049d581042_isr_945755_57.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372 parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373 parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash 07049c6b6fa9575b0922a8a9ea80d8ff parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/cff5b11a-a171-4fc5-8755-d3e63d70140b_isr_945867_8.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/cff5b11a-a171-4fc5-8755-d3e63d70140b_isr_945867_8.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375 parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376 parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 4ad16655322fb26d19dfe54be1f28c1c parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9d786e71-b5a1-475a-82e3-e197d30b476f_isr_945868_6.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9d786e71-b5a1-475a-82e3-e197d30b476f_isr_945868_6.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378 parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379 parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 0996963feb0d7af671e2900dd4ae595c parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/cb6474f6-96bd-4d7b-b469-e5ccc67af65e_isr_945755_6.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/cb6474f6-96bd-4d7b-b469-e5ccc67af65e_isr_945755_6.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381 parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382 parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 25b5151793102a7e67adfc177caeb37f parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/91a4e7a9-2c33-4bfc-821c-ec43e7c63a6a_isr_945867_11.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/91a4e7a9-2c33-4bfc-821c-ec43e7c63a6a_isr_945867_11.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384 parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385 parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash da0412728287969519f8840885b19497 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/b8c27e31-c304-4891-976c-cd625abb55bc_isr_945869_30.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/b8c27e31-c304-4891-976c-cd625abb55bc_isr_945869_30.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387 parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388 parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash a637c1a3892c824b4257cbc8137163d4 parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/60825274-930c-48a1-8a2e-ebce681bc557_isr_945756_46.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/60825274-930c-48a1-8a2e-ebce681bc557_isr_945756_46.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390 parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391 parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 05ddd63125af1d9a46c95f3bc9f1278b parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 132 parsl.dataflow.strategy DEBUG: Executor multi has 132 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 32 more blocks parsl.executors.status_handling INFO: Scaling out by 32 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-13 18:33:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-13 18:33:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 18:33:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0cf2a0db-9575-49bc-a01f-2bb29683cc96_isr_945756_5.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0cf2a0db-9575-49bc-a01f-2bb29683cc96_isr_945756_5.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393 parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394 parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 7df946c0265cee6b357f163af3c4bcd8 parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/87bada8d-8e07-4292-9985-c16920ed3dce_isr_945868_51.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/87bada8d-8e07-4292-9985-c16920ed3dce_isr_945868_51.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396 parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397 parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash dd51c2c7d472fdcaea3282c9117f2460 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f0bb07d6-28af-4a88-8ccb-b0578fc56e05_isr_945869_24.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f0bb07d6-28af-4a88-8ccb-b0578fc56e05_isr_945869_24.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399 parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400 parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash e88cc7e5c6f5ce4b95007f858478b347 parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/41cdf9ba-bf9b-42b6-bac3-da11d15614c0_isr_945867_18.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/41cdf9ba-bf9b-42b6-bac3-da11d15614c0_isr_945867_18.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402 parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403 parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash 893f5aa3b50ad0ca100079ebe368f97b parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0c571b4c-76ac-4b14-a7ff-2c05c03f99e8_isr_945756_50.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0c571b4c-76ac-4b14-a7ff-2c05c03f99e8_isr_945756_50.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405 parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406 parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash e800d068bcd0d980c8639d5aa9f154f6 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/7892dd7e-24f0-4007-9426-87c172610a3d_isr_945755_4.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/7892dd7e-24f0-4007-9426-87c172610a3d_isr_945755_4.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408 parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409 parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash c1fceee7aee9c8a321949774f873873e parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/d01f04e9-fad9-43ca-96bd-a3e86846dc11_isr_945756_29.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/d01f04e9-fad9-43ca-96bd-a3e86846dc11_isr_945756_29.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411 parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash f63ffac279b90bc5c4b55c468e052182 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d3f0963b-4b13-4185-9ec5-1b916b61463c_isr_945757_61.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d3f0963b-4b13-4185-9ec5-1b916b61463c_isr_945757_61.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414 parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415 parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 560c25a7f4f7a6d419031164bcf9fb54 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0330dcdc-d82c-4d96-bf8f-69a76d7f75f3_isr_945756_56.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0330dcdc-d82c-4d96-bf8f-69a76d7f75f3_isr_945756_56.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417 parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418 parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash b4e27bb48c7ba7382bdb6f7045bca157 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/bcb05fb3-222a-45f8-97da-835483d65abc_isr_945755_19.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/bcb05fb3-222a-45f8-97da-835483d65abc_isr_945755_19.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420 parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421 parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash dd2c173b8e1e301851522316f5052908 parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/72bf8367-6ac4-408c-8512-29ed514b9632_isr_945868_8.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/72bf8367-6ac4-408c-8512-29ed514b9632_isr_945868_8.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423 parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424 parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash f91e5d79dddf1ca871aeedfc74357252 parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8b4b0938-357d-4d01-9d15-618d45da728c_isr_945867_33.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8b4b0938-357d-4d01-9d15-618d45da728c_isr_945867_33.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426 parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427 parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash f8e0a2549ceb80715d2f4503b652225c parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/829476d6-d576-43ea-b12e-9fc1d8f51348_isr_945868_32.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/829476d6-d576-43ea-b12e-9fc1d8f51348_isr_945868_32.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429 parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430 parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash de854ddb39582f910f6963c7b9b7e866 parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/f8b7576c-d600-48c1-b0ac-f3ac20af6c7a_isr_945867_13.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/f8b7576c-d600-48c1-b0ac-f3ac20af6c7a_isr_945867_13.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432 parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433 parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash 5d468d2f563c27544bad8f20f0592445 parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/3abdb6f9-03ad-44a6-a5e0-f7f5f46b1d88_isr_945869_13.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/3abdb6f9-03ad-44a6-a5e0-f7f5f46b1d88_isr_945869_13.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436 parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash c29d7900293f1c44604c74c76e2efc1b parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/87f34727-610d-4d86-89ac-84359660e677_isr_945869_57.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/87f34727-610d-4d86-89ac-84359660e677_isr_945869_57.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438 parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439 parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 1d112bc92a1189e017590db2f41b22db parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e990570a-3981-4d31-80f0-49763a857108_isr_945869_12.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e990570a-3981-4d31-80f0-49763a857108_isr_945869_12.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441 parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442 parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash 52f585708c2fc6a5366109278b25952a parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/69067d16-d94c-4d38-a412-8e6f98c660f8_isr_945867_39.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/69067d16-d94c-4d38-a412-8e6f98c660f8_isr_945867_39.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444 parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445 parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 149ed09d38fcb41429a65d7a61776955 parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c65167d5-83c3-4560-9396-98b28ea9efa2_isr_945868_52.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c65167d5-83c3-4560-9396-98b28ea9efa2_isr_945868_52.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447 parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448 parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 32c3ffe342c35f6e1c52a25acdc6af5b parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/bec9c9ab-1d70-4190-aaf6-89807b11962a_isr_945867_43.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/bec9c9ab-1d70-4190-aaf6-89807b11962a_isr_945867_43.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450 parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451 parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 9bca3d7be065a9c173d6536a4ae7b34d parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/bc0caf00-b0c9-4076-bd44-0b297772fefc_isr_945756_24.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/bc0caf00-b0c9-4076-bd44-0b297772fefc_isr_945756_24.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453 parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454 parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash a5cfe13743c89a427ff25810cd6084d3 parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/ac0e44e0-f40e-4aaa-811e-bc1f9c48f561_isr_945757_29.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/ac0e44e0-f40e-4aaa-811e-bc1f9c48f561_isr_945757_29.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456 parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457 parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash cb5682f9c839936443e182b6c20d1621 parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492464000e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/29848de2-11e2-445f-b0e6-be6201e607d9_isr_945756_7.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/29848de2-11e2-445f-b0e6-be6201e607d9_isr_945756_7.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459 parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460 parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash ac92d6369ea0308e726ef66c3752fd15 parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246400400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/34c96422-0d9f-405c-a88d-f240882c3ff2_isr_945869_35.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/34c96422-0d9f-405c-a88d-f240882c3ff2_isr_945869_35.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462 parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463 parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash f47af81d18ad704e95902ebb31b3c907 parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246400720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/44c25584-b173-452e-9cdd-96eb793aa963_isr_945868_1.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/44c25584-b173-452e-9cdd-96eb793aa963_isr_945868_1.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465 parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466 parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 8472db3f4d13d5dd03ae4a4cdc2e4e83 parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246400a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/dfd89499-cf54-48b7-9763-b5ba26d60cfd_isr_945869_52.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/dfd89499-cf54-48b7-9763-b5ba26d60cfd_isr_945869_52.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468 parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469 parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash 642667704e2854ac92066ecc0411fbe8 parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246400d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/2f32c7f2-41a5-49c4-bb89-fcc701e874ca_isr_945868_33.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/2f32c7f2-41a5-49c4-bb89-fcc701e874ca_isr_945868_33.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471 parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472 parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash e8e4360543e2cad05ad6e8073b3a55cd parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8930ca1a-fd73-4c03-bee0-0d524622b2c5_isr_945869_40.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8930ca1a-fd73-4c03-bee0-0d524622b2c5_isr_945869_40.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474 parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475 parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash 6274096010e0334450993a4bb896d239 parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492464013a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 18:33:37 proc_lsst.multi:170 [INFO] [multi] job_id 17020626 proc_lsst.multi INFO: [multi] job_id 17020626 2024-03-13 18:33:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020626 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020626 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17020626 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-13 18:33:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-13 18:33:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/abc00f5e-a860-46d1-ae98-47416992e45c_isr_945869_49.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/abc00f5e-a860-46d1-ae98-47416992e45c_isr_945869_49.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477 parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478 parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 8d21761e42fddc5a30502c43b5fa9d45 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8b35d30e-8a67-431c-8071-235cf3a7e210_isr_945755_38.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8b35d30e-8a67-431c-8071-235cf3a7e210_isr_945755_38.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480 parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash ded97aa74dbf3f1158d9e0b06cb1648a parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/ab6d557c-5609-41e6-b7b2-3126f187fbba_isr_945867_41.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/ab6d557c-5609-41e6-b7b2-3126f187fbba_isr_945867_41.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483 parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484 parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash 60a0ad819da6a5430430433a7b2e3736 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c589d8cd-717a-4efe-824f-6b9812178af9_isr_945868_47.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c589d8cd-717a-4efe-824f-6b9812178af9_isr_945868_47.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486 parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487 parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 4ac172261c04e9ddf3199f04f6ef3e3c parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/332adf15-eee1-4bfa-aa10-d6f3f70f7b40_isr_945867_14.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/332adf15-eee1-4bfa-aa10-d6f3f70f7b40_isr_945867_14.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489 parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490 parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 4e9424c99046dea24a793262dc3cbf49 parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/580272e9-d943-4f2f-b557-1c299fb1f992_isr_945756_34.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/580272e9-d943-4f2f-b557-1c299fb1f992_isr_945756_34.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492 parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493 parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 47170bfe6a3cff3eb0a571da283c1b7a parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4c58c91d-ef29-4930-9fe7-4832ba83a993_isr_945867_59.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4c58c91d-ef29-4930-9fe7-4832ba83a993_isr_945867_59.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495 parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496 parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash 3d2eeb056d22d7eb9578f32b6ec48541 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/3c9226f4-5f46-4346-89b5-d36aa7c530ea_isr_945756_48.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/3c9226f4-5f46-4346-89b5-d36aa7c530ea_isr_945756_48.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498 parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499 parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 73771417257278fdf4339d305913df6e parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/690a7304-6757-4d5e-9071-af237c446826_isr_945867_22.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/690a7304-6757-4d5e-9071-af237c446826_isr_945867_22.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501 parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502 parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash fd7b6d348e51dd31a9e19696d030c15f parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246402660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/41ae89f9-fa86-4190-9d80-f101343326af_isr_945869_22.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/41ae89f9-fa86-4190-9d80-f101343326af_isr_945869_22.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505 parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash e5f216b8336669b2341240a597f07e23 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246402a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/6ec1231c-a89f-4c8e-b18c-6f8c536a5eba_isr_945868_44.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/6ec1231c-a89f-4c8e-b18c-6f8c536a5eba_isr_945868_44.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507 parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508 parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 85026505e44db6cb41e0d731b5a8195b parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246402b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/224885b0-1295-46d7-bd02-5c37d6344487_isr_945755_10.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/224885b0-1295-46d7-bd02-5c37d6344487_isr_945755_10.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510 parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511 parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash 8fff4d5dc31a2e03831398539e5b9cbb parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246402e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/fba0e134-a464-40a3-a369-a2d16ab125f1_isr_945756_43.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/fba0e134-a464-40a3-a369-a2d16ab125f1_isr_945756_43.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513 parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514 parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 5bd8858fffaced94cd23c45178192161 parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492464032e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/823551f6-139c-42cf-acb5-8b1922ee8cb2_isr_945757_36.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/823551f6-139c-42cf-acb5-8b1922ee8cb2_isr_945757_36.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516 parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517 parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash b41b50cb66ff9cf7598a277702787e7c parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246403600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/44b7a6eb-9e02-4470-b723-f80d538ca7c0_isr_945756_44.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/44b7a6eb-9e02-4470-b723-f80d538ca7c0_isr_945756_44.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519 parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520 parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 88719fe47282c5cc86b78f33b1305873 parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246403880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/43e82438-4d83-4843-bf88-417af9b0792a_isr_945869_16.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/43e82438-4d83-4843-bf88-417af9b0792a_isr_945869_16.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522 parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523 parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash c6136ae629d03253ed265920e17fd02e parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246403d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/85c74c79-959c-47bc-8e17-adc11942655c_isr_945867_25.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/85c74c79-959c-47bc-8e17-adc11942655c_isr_945867_25.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525 parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526 parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash 42322d6ed2391737db665892b8b510bc parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/2a3e3c86-cea9-42ed-b2d6-7d1235febbeb_isr_945868_25.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/2a3e3c86-cea9-42ed-b2d6-7d1235febbeb_isr_945868_25.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528 parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529 parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash d323f95c2f350767bc0b78e13be60649 parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/3edfc5c7-ca73-40c4-9455-b6b76f231d45_isr_945869_1.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/3edfc5c7-ca73-40c4-9455-b6b76f231d45_isr_945869_1.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531 parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532 parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash deb2537ea30c7a87fba47c708440f69f parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/eae70657-8f36-47f4-a892-9645af59539a_isr_945757_27.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/eae70657-8f36-47f4-a892-9645af59539a_isr_945757_27.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534 parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535 parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash a29fe4330693b2c2cad2aee6e8fba59c parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/d01630ea-2d5d-452f-81af-57a2b04ef1d8_isr_945867_34.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/d01630ea-2d5d-452f-81af-57a2b04ef1d8_isr_945867_34.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537 parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538 parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash cab6039322f793980a91cfaf53d92d85 parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/6da22f22-b0e6-4e1e-8905-ba2cb0e1fa7a_isr_945755_12.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/6da22f22-b0e6-4e1e-8905-ba2cb0e1fa7a_isr_945755_12.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540 parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541 parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 03e3aaa86d6dc7bf1c69e893c6ceb9ff parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d2ccce26-978a-446e-b1a2-2ce8fe701fe0_isr_945757_21.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d2ccce26-978a-446e-b1a2-2ce8fe701fe0_isr_945757_21.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543 parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544 parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash a6a90a47eefbae86da3943951b5d63aa parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/46c14efe-5889-4c4a-b614-5e338201c125_isr_945757_37.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/46c14efe-5889-4c4a-b614-5e338201c125_isr_945757_37.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546 parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547 parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 279c4a891f1c49cf1564599cacef2567 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/a3bf4994-006e-428a-907a-69d4d57b072c_isr_945755_56.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/a3bf4994-006e-428a-907a-69d4d57b072c_isr_945755_56.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549 parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 4447c39c356619865263f6eb861dd81b parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/144763f3-c7f2-4def-a169-7d14727eb42d_isr_945867_58.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/144763f3-c7f2-4def-a169-7d14727eb42d_isr_945867_58.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552 parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553 parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash 6045028373dca2951a4c0444732778b9 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8a936f05-2edd-4128-89fe-1720f61c3a4d_isr_945867_12.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8a936f05-2edd-4128-89fe-1720f61c3a4d_isr_945867_12.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555 parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556 parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash 9d1408991b94d01d6a3bdce6cd146734 parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4e507d82-3fbd-47fc-b365-29418154e3bb_isr_945867_60.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4e507d82-3fbd-47fc-b365-29418154e3bb_isr_945867_60.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558 parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559 parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash a2f042f53a539e883fef9cf07e6e665f parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/ba4bb611-8169-4edb-a78f-37d6fc108caf_isr_945757_33.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/ba4bb611-8169-4edb-a78f-37d6fc108caf_isr_945757_33.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561 parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562 parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 70cd231845cb2ab6f7ebeb7e582486bb parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8c845107-2787-4ced-bb15-08fa1d0fec2d_isr_945869_55.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8c845107-2787-4ced-bb15-08fa1d0fec2d_isr_945869_55.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564 parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565 parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 862e2baf268b106e0f4af84ad9e833e6 parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/c7348a38-4dd3-47a7-bb3e-91ae8c5f8e52_isr_945867_32.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/c7348a38-4dd3-47a7-bb3e-91ae8c5f8e52_isr_945867_32.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567 parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568 parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 869745d9e92ce4198c55ecbc1faedcc4 parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/17308fdb-e39e-47a7-afc7-2ac0e2959e8f_isr_945756_33.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/17308fdb-e39e-47a7-afc7-2ac0e2959e8f_isr_945756_33.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570 parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571 parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash cebd7a0da11d12e249302cdc9ab2e68b parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/186a4c82-d3bd-4f41-a392-7423820519ca_isr_945756_15.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/186a4c82-d3bd-4f41-a392-7423820519ca_isr_945756_15.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573 parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574 parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash e89360b6e63ece8e7f98d1324cd5eaf9 parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8f430c6f-270d-46b9-95e3-015b6717eded_isr_945868_26.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8f430c6f-270d-46b9-95e3-015b6717eded_isr_945868_26.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576 parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577 parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash e773729eb2060b81b983b05829ca48d2 parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/66cdd3b2-693b-4bab-ab1d-df1a8b30f66b_isr_945868_10.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/66cdd3b2-693b-4bab-ab1d-df1a8b30f66b_isr_945868_10.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579 parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580 parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 47595e3f957c429e05dd61a0711ff577 parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246284180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/f69bf965-ec06-4c78-bbe0-fe6ec17bc80a_isr_945755_49.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/f69bf965-ec06-4c78-bbe0-fe6ec17bc80a_isr_945755_49.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582 parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583 parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 56d58f88e163720abdd4a061bf8fad81 parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462844a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/468d0b31-3e4c-4959-b51b-20b43a321130_isr_945757_31.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/468d0b31-3e4c-4959-b51b-20b43a321130_isr_945757_31.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585 parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586 parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash 89f62d2afc0b005a28372c6589a95a8b parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462847c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/460cfe67-c4af-4320-a996-5ab83531d2fc_isr_945757_9.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/460cfe67-c4af-4320-a996-5ab83531d2fc_isr_945757_9.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588 parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589 parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 159b7237a592e315b46af9dbe4942d6b parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246284ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/480e777a-a5a3-4538-9132-f65ae41fb23d_isr_945869_45.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/480e777a-a5a3-4538-9132-f65ae41fb23d_isr_945869_45.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591 parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592 parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 42c0ebb6ea30b62d2cba2262df6cbc2f parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246284e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/59a979ad-03ad-46eb-995f-4d1864699ca1_isr_945756_8.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/59a979ad-03ad-46eb-995f-4d1864699ca1_isr_945756_8.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594 parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595 parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash ed9a6c6238ed3e61f55991df1a10bbb5 parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246285120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/9bcedbd0-bc83-464e-816b-55e4ee76a180_isr_945869_6.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/9bcedbd0-bc83-464e-816b-55e4ee76a180_isr_945869_6.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597 parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598 parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash dbd3fbcaad4008d62d509da93eb79bd4 parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246285440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 18:33:37 proc_lsst.multi:170 [INFO] [multi] job_id 17020627 proc_lsst.multi INFO: [multi] job_id 17020627 2024-03-13 18:33:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 18:33:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020627 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020627 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17020627 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-13 18:33:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-13 18:33:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 18:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/76d57c1d-8272-43a2-bf85-0e0ea65c06e8_isr_945757_3.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/76d57c1d-8272-43a2-bf85-0e0ea65c06e8_isr_945757_3.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600 parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601 parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 6dd268a60c3fa73e768b3c6effe7ae83 parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462854e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/18f9268d-86ba-46f2-b7f6-40399a7af489_isr_945869_11.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/18f9268d-86ba-46f2-b7f6-40399a7af489_isr_945869_11.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603 parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604 parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash cdbac11bab41c24a574f91148cb93a6d parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246285940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3c4bda6c-36e9-426d-8b24-528e0fe608af_isr_945755_29.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3c4bda6c-36e9-426d-8b24-528e0fe608af_isr_945755_29.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606 parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607 parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash d2233b11b0be035c707556c79628518b parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246285c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3c75de45-b7b5-4d23-810b-45793d590b03_isr_945755_22.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3c75de45-b7b5-4d23-810b-45793d590b03_isr_945755_22.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609 parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610 parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 3264e6bcb6c2aa1d53cc0d0a556cf664 parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246285ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/54088db6-7a7a-4eb9-af38-f307cd031650_isr_945756_2.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/54088db6-7a7a-4eb9-af38-f307cd031650_isr_945756_2.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612 parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613 parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash 581cbbfd7f7072046a84f666aba8c00a parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/4cae27c9-96a4-4939-b84a-3233e38f6265_isr_945756_57.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/4cae27c9-96a4-4939-b84a-3233e38f6265_isr_945756_57.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615 parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616 parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash 2fa296fdad75ecccd5ba53616226feab parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/0ab6ede6-03e9-4788-9689-5df7f4737801_isr_945869_34.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/0ab6ede6-03e9-4788-9689-5df7f4737801_isr_945869_34.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618 parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619 parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 8599cbab2ecc25ab73c6d4824461e0e5 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/9f47f12d-d9b0-4199-8384-93a859efdf5b_isr_945755_21.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/9f47f12d-d9b0-4199-8384-93a859efdf5b_isr_945755_21.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621 parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622 parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash dabb3e97728a9059d671768853b697bf parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/6405c211-b0a4-49bf-8139-e20e47ad1e3c_isr_945756_30.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/6405c211-b0a4-49bf-8139-e20e47ad1e3c_isr_945756_30.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624 parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625 parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 0162f74766389308b30a0e5eb90231e0 parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/c972aef3-1931-4d45-b088-073be10f39f5_isr_945756_22.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/c972aef3-1931-4d45-b088-073be10f39f5_isr_945756_22.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627 parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628 parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash 1c92d50a6e9b5ea08ae75136b78c5136 parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1cd2e6ae-4dfe-48fb-acc6-5f2a2a845483_isr_945867_1.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1cd2e6ae-4dfe-48fb-acc6-5f2a2a845483_isr_945867_1.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630 parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631 parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash 9621a5b5d123c40c73d24661356f55dc parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/7cd86ec1-e970-4225-b864-9d537c473b1b_isr_945868_5.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/7cd86ec1-e970-4225-b864-9d537c473b1b_isr_945868_5.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633 parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634 parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 4dceaa48692d5327a4236b8c5ca87333 parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462872e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/13257298-71c7-4af5-a9dd-2b7b7335bcb7_isr_945755_9.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/13257298-71c7-4af5-a9dd-2b7b7335bcb7_isr_945755_9.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636 parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637 parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash 26cef781469d527deb0d874b555264a2 parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246287600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/6e8992ec-c889-4bb6-84f2-445ff625cea8_isr_945867_49.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/6e8992ec-c889-4bb6-84f2-445ff625cea8_isr_945867_49.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639 parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640 parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash c993509658107e92d1014fff56f0323a parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246287880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8185400d-1be4-426b-a705-c4e14c785100_isr_945755_60.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8185400d-1be4-426b-a705-c4e14c785100_isr_945755_60.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642 parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643 parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash f18e3c2d750aae7efe0f9bb51707a2e7 parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246287d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/eba12411-e3ec-48cc-8a5f-be3a65ae55de_isr_945756_14.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/eba12411-e3ec-48cc-8a5f-be3a65ae55de_isr_945756_14.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645 parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646 parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash 1112d4de0f250afcb1849320fa4ccee1 parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/6ea841dc-e970-4156-a676-a780d923c1b0_isr_945868_36.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/6ea841dc-e970-4156-a676-a780d923c1b0_isr_945868_36.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648 parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649 parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash 98e274d58b0e8c522ca50fd9cb750338 parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/636ef77a-6355-462b-8b8c-d6721212e316_isr_945755_5.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/636ef77a-6355-462b-8b8c-d6721212e316_isr_945755_5.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651 parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652 parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash 137722b0c913efb8887fd38a2e40b9a6 parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f7aa67c1-b06d-470d-ab69-020a5ed2e3e5_isr_945757_19.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f7aa67c1-b06d-470d-ab69-020a5ed2e3e5_isr_945757_19.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654 parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655 parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash ffb5cc0c78919b531f651029985236ff parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/5e64e9f1-0a44-4d70-830b-87a530baf0f8_isr_945756_53.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/5e64e9f1-0a44-4d70-830b-87a530baf0f8_isr_945756_53.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657 parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658 parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash 5f5130620304c94be52acc89c7c18a77 parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/ccfc3bcb-be79-4663-968d-183d61a5794b_isr_945757_38.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/ccfc3bcb-be79-4663-968d-183d61a5794b_isr_945757_38.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660 parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661 parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash 4f7f8f8c49013b31c81416eea90777b2 parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/f1960032-e7fe-409d-a319-a17103452f72_isr_945867_10.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/f1960032-e7fe-409d-a319-a17103452f72_isr_945867_10.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663 parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664 parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash c2d2008ac36b28f6a0d051ce1e8d56d7 parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/96784ad5-e339-43c2-93f1-2a78614f5666_isr_945755_45.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/96784ad5-e339-43c2-93f1-2a78614f5666_isr_945755_45.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666 parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667 parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash 69099a3238d916383fa0c2232ca62c6e parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/a63c1ac2-f9aa-40b8-8f70-f1a48c0d6eb5_isr_945868_38.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/a63c1ac2-f9aa-40b8-8f70-f1a48c0d6eb5_isr_945868_38.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669 parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670 parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash 00d7dc005bb2d604325426fc5fd6bdcf parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/b1c55525-4144-41a9-82df-6d2fd9e9672f_isr_945868_20.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/b1c55525-4144-41a9-82df-6d2fd9e9672f_isr_945868_20.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672 parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673 parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash 3852bc7ff93815730df20f6cacd50a0f parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462be2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/b72091e8-354e-4c28-ab9b-ca0e905ff338_isr_945869_31.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/b72091e8-354e-4c28-ab9b-ca0e905ff338_isr_945869_31.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675 parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676 parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 144b2844c4add818fefb5ff5115a1d38 parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462be5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/a219b5d4-d320-4cef-8055-873221230d17_isr_945868_40.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/a219b5d4-d320-4cef-8055-873221230d17_isr_945868_40.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678 parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679 parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash 2a4e3a58302793befd95dbec10348582 parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462be8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/58922fbf-1fff-4db5-b7d1-bab2351756f9_isr_945755_27.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/58922fbf-1fff-4db5-b7d1-bab2351756f9_isr_945755_27.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681 parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682 parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash 62f5df575e3d18c3ff6480a740cb8fd4 parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/94e83da6-1063-45f6-abd3-fd752121b4e9_isr_945868_62.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/94e83da6-1063-45f6-abd3-fd752121b4e9_isr_945868_62.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684 parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685 parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 695616e333c0a6f6feb3b24f9caaa100 parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/9ec1ef08-3848-481b-a922-d0eaf7006bc4_isr_945869_32.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/9ec1ef08-3848-481b-a922-d0eaf7006bc4_isr_945869_32.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687 parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688 parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash c80f40e617716d14c10708ed7f0aa166 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/ce353005-137b-484d-b822-0d740f982a28_isr_945756_12.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/ce353005-137b-484d-b822-0d740f982a28_isr_945756_12.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690 parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691 parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash a223f80d132423025958c3272bb52f44 parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f0f932a9-5280-41b9-91e6-699b77217ec7_isr_945757_46.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f0f932a9-5280-41b9-91e6-699b77217ec7_isr_945757_46.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693 parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694 parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash f52e430944385c09c32ab4db6e822752 parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/45856bf5-c4e5-4db6-ab92-444bee26df11_isr_945755_53.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/45856bf5-c4e5-4db6-ab92-444bee26df11_isr_945755_53.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696 parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697 parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash 190dcdee71367cafe1f8c41dcf10f421 parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d6396833-efbe-49bb-b036-a6090ecd76c5_isr_945868_34.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d6396833-efbe-49bb-b036-a6090ecd76c5_isr_945868_34.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699 parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700 parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash d545e92ec678cbe9b7c416701cf7dc1c parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/32e75895-cf20-43f9-813b-d765bcc90502_isr_945757_11.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/32e75895-cf20-43f9-813b-d765bcc90502_isr_945757_11.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702 parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703 parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash b75520d35045bd824781c6505a9b4293 parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246310220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 18:33:37 proc_lsst.multi:170 [INFO] [multi] job_id 17020628 proc_lsst.multi INFO: [multi] job_id 17020628 2024-03-13 18:33:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 18:33:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020628 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020628 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17020628 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-13 18:33:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-13 18:33:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 18:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0d2c3011-2a86-4e49-9293-56033ab02a8e_isr_945756_18.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0d2c3011-2a86-4e49-9293-56033ab02a8e_isr_945756_18.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705 parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706 parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 6a69e8612de3734ce331e0d5d9ff86e2 parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463102c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2a0ed1c6-f1fc-4fa3-9ecf-62226db517b8_isr_945756_41.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2a0ed1c6-f1fc-4fa3-9ecf-62226db517b8_isr_945756_41.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708 parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709 parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 295ac273a5a2c5680a7838057949db83 parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246310680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d2684aa5-1c2f-4292-91de-a0aa9ae4582e_isr_945755_61.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d2684aa5-1c2f-4292-91de-a0aa9ae4582e_isr_945755_61.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711 parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712 parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 16ea903d45bf9b9e7215238a8ada17a3 parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463109a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/28f74549-c3dd-49fa-90ae-e84ca71d1d2c_isr_945868_3.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/28f74549-c3dd-49fa-90ae-e84ca71d1d2c_isr_945868_3.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714 parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715 parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 04cb5e9026cfa140b47e614d1ea37fb7 parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246310d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d7c56008-c0d8-4be2-b735-ac62b6e16dc0_isr_945755_26.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d7c56008-c0d8-4be2-b735-ac62b6e16dc0_isr_945755_26.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717 parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718 parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash 4893f16b3dbcbaedf97231b4c15aa7fe parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246311080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/434ca6db-14af-4874-8c1f-5ba790c0f95d_isr_945869_3.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/434ca6db-14af-4874-8c1f-5ba790c0f95d_isr_945869_3.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720 parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721 parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash fc6ec2f8158a164d5a994ccd67d2de3e parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463113a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/74ad9c98-feb6-43ab-bca3-068a016aa798_isr_945755_35.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/74ad9c98-feb6-43ab-bca3-068a016aa798_isr_945755_35.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723 parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724 parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash 95d789efe74f6b11dd58951d19ad58bd parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246311620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/46ca6f67-28d6-4a25-bc90-768dea4203dc_isr_945757_6.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/46ca6f67-28d6-4a25-bc90-768dea4203dc_isr_945757_6.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726 parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727 parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash 85f5e8030cfe3a21427a9ff48ca7d6b7 parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246311940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/231f7a77-be12-4292-a8ee-527af7270885_isr_945869_50.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/231f7a77-be12-4292-a8ee-527af7270885_isr_945869_50.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729 parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730 parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash 6342ca7202dc79bad0210c33f0987522 parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246311c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/57d3e197-d16b-47b9-b014-aeaaf17cf744_isr_945867_16.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/57d3e197-d16b-47b9-b014-aeaaf17cf744_isr_945867_16.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732 parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733 parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash 83d473bacce1d4cedc4bf97a036de2b2 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246345440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/b7bd6a74-ee93-4073-bdb2-8faddc43495c_isr_945869_33.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/b7bd6a74-ee93-4073-bdb2-8faddc43495c_isr_945869_33.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735 parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736 parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash 82774017d228b4d4fbffd4526ead425c parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246345120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/a95707ef-0303-4975-8e84-60395aff6159_isr_945755_58.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/a95707ef-0303-4975-8e84-60395aff6159_isr_945755_58.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738 parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739 parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash d5909439c3adc5332c6432008516bf3b parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1fdd0d15-b4aa-4bdd-a30b-04cbba44063a_isr_945867_20.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1fdd0d15-b4aa-4bdd-a30b-04cbba44063a_isr_945867_20.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741 parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742 parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 744 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 744 has memoization hash c1950c646a6a74313b71401cc8cfe3a0 parsl.dataflow.memoization INFO: Task 744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/69549d2a-acc6-4939-8b1a-0a31cf46434c_isr_945756_52.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/69549d2a-acc6-4939-8b1a-0a31cf46434c_isr_945756_52.stderr parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 745 submitted for App characterizeImage, waiting on task 744 parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 745 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 746 submitted for App calibrate, waiting on task 745 parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 746 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 747 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 747 has memoization hash 5ce16e88ef3a7709ae26f771b9ddb989 parsl.dataflow.memoization INFO: Task 747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246312520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/f33c0f00-bd9b-4060-94a7-93e857e7f9f2_isr_945867_53.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/f33c0f00-bd9b-4060-94a7-93e857e7f9f2_isr_945867_53.stderr parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 748 submitted for App characterizeImage, waiting on task 747 parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 748 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 749 submitted for App calibrate, waiting on task 748 parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 749 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 750 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 750 has memoization hash 20d822a3d79e122020d29bf6aaacf302 parsl.dataflow.memoization INFO: Task 750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463128e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/cbc870ed-fd0d-44e1-9647-7820621835f9_isr_945868_18.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/cbc870ed-fd0d-44e1-9647-7820621835f9_isr_945868_18.stderr parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 751 submitted for App characterizeImage, waiting on task 750 parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 751 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 752 submitted for App calibrate, waiting on task 751 parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 752 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 753 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 753 has memoization hash cc4d65a324c14fcb6ee53702b2b87575 parsl.dataflow.memoization INFO: Task 753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246312a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/169a4eb0-d14e-4ddd-9423-37d8d1dcf650_isr_945756_37.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/169a4eb0-d14e-4ddd-9423-37d8d1dcf650_isr_945756_37.stderr parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 754 submitted for App characterizeImage, waiting on task 753 parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 754 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 755 submitted for App calibrate, waiting on task 754 parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 755 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 756 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 756 has memoization hash 0f65420212237dbcaa93e9013e317045 parsl.dataflow.memoization INFO: Task 756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246312d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/940396b7-6cae-4d80-b309-0c2ddae0f919_isr_945867_47.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/940396b7-6cae-4d80-b309-0c2ddae0f919_isr_945867_47.stderr parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 757 submitted for App characterizeImage, waiting on task 756 parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 757 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 758 submitted for App calibrate, waiting on task 757 parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 758 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 759 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 759 has memoization hash 9581b3a55deb4d488a7e02a116042ed9 parsl.dataflow.memoization INFO: Task 759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463131a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/c7f1cef7-3ccc-489b-92a5-f9d91a4fd039_isr_945756_59.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/c7f1cef7-3ccc-489b-92a5-f9d91a4fd039_isr_945756_59.stderr parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 760 submitted for App characterizeImage, waiting on task 759 parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 760 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 761 submitted for App calibrate, waiting on task 760 parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 761 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 762 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 762 has memoization hash ef6d3ab84880e7d58722c8e8fd7ccbb2 parsl.dataflow.memoization INFO: Task 762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463134c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/606249f0-2a0d-4dfd-be4b-7ee79632e543_isr_945868_2.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/606249f0-2a0d-4dfd-be4b-7ee79632e543_isr_945868_2.stderr parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 763 submitted for App characterizeImage, waiting on task 762 parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 763 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 764 submitted for App calibrate, waiting on task 763 parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 764 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 765 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 765 has memoization hash 51580768b7049e1f199a67acadeb3016 parsl.dataflow.memoization INFO: Task 765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246313740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b77c33c5-1683-4103-bf98-a34beec564ff_isr_945755_25.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b77c33c5-1683-4103-bf98-a34beec564ff_isr_945755_25.stderr parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 766 submitted for App characterizeImage, waiting on task 765 parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 766 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 767 submitted for App calibrate, waiting on task 766 parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 767 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 768 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 768 has memoization hash 51da62051c1c633368590ed86130b695 parsl.dataflow.memoization INFO: Task 768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246313c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/4516ff7a-d29f-425e-b05e-978e21604a4b_isr_945756_42.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/4516ff7a-d29f-425e-b05e-978e21604a4b_isr_945756_42.stderr parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 769 submitted for App characterizeImage, waiting on task 768 parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 769 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 770 submitted for App calibrate, waiting on task 769 parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 770 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 771 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 771 has memoization hash 196e0a16b2480100069c01d26de55e9f parsl.dataflow.memoization INFO: Task 771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246313ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ef4dd465-f58b-4ddb-be4b-96ca43364e3b_isr_945868_22.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ef4dd465-f58b-4ddb-be4b-96ca43364e3b_isr_945868_22.stderr parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 772 submitted for App characterizeImage, waiting on task 771 parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 772 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 773 submitted for App calibrate, waiting on task 772 parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 773 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 774 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 774 has memoization hash 87c28e0e68cbdad4de9758fcda54b841 parsl.dataflow.memoization INFO: Task 774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/204a8a28-8805-4296-ab51-16910c3c489c_isr_945756_21.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/204a8a28-8805-4296-ab51-16910c3c489c_isr_945756_21.stderr parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 775 submitted for App characterizeImage, waiting on task 774 parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 775 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 776 submitted for App calibrate, waiting on task 775 parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 776 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 777 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 777 has memoization hash a84b6bead43c8ff189ab99561bd4f8a2 parsl.dataflow.memoization INFO: Task 777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463447c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e0c6f304-709c-4ea4-93e2-aa099718b803_isr_945755_43.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e0c6f304-709c-4ea4-93e2-aa099718b803_isr_945755_43.stderr parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 778 submitted for App characterizeImage, waiting on task 777 parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 778 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 779 submitted for App calibrate, waiting on task 778 parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 779 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 780 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 780 has memoization hash 8ebe73dfe374f83810e8e8972045c719 parsl.dataflow.memoization INFO: Task 780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463444a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/a4a82628-37e8-4109-b9d7-c0bb33af44e9_isr_945867_44.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/a4a82628-37e8-4109-b9d7-c0bb33af44e9_isr_945867_44.stderr parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 781 submitted for App characterizeImage, waiting on task 780 parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 781 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 782 submitted for App calibrate, waiting on task 781 parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 782 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 783 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 783 has memoization hash d9d7c8356f234e83dad7299643efad18 parsl.dataflow.memoization INFO: Task 783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/50d3b6a0-e9d8-43e0-adb3-9eb2df537649_isr_945869_28.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/50d3b6a0-e9d8-43e0-adb3-9eb2df537649_isr_945869_28.stderr parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 784 submitted for App characterizeImage, waiting on task 783 parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 784 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 785 submitted for App calibrate, waiting on task 784 parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 785 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 786 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 786 has memoization hash 6f39b10cfb0765bad502e4185784baf1 parsl.dataflow.memoization INFO: Task 786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463456c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/63c1a3e7-7bf1-41ff-b630-be601a4b2554_isr_945869_44.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/63c1a3e7-7bf1-41ff-b630-be601a4b2554_isr_945869_44.stderr parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 787 submitted for App characterizeImage, waiting on task 786 parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 787 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 788 submitted for App calibrate, waiting on task 787 parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 788 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 789 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 789 has memoization hash 39cd4782b2f1c6a88f9dc3177298e723 parsl.dataflow.memoization INFO: Task 789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463459e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2db68b08-6f0c-4425-b417-a399a9527448_isr_945756_1.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2db68b08-6f0c-4425-b417-a399a9527448_isr_945756_1.stderr parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 790 submitted for App characterizeImage, waiting on task 789 parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 790 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 791 submitted for App calibrate, waiting on task 790 parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 791 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 792 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash b3a9ee2772b27306801bcfe92142820f parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246345d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d548272e-48ee-46d3-aa4c-4d59948b0eb4_isr_945868_21.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d548272e-48ee-46d3-aa4c-4d59948b0eb4_isr_945868_21.stderr parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 793 submitted for App characterizeImage, waiting on task 792 parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 793 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 794 submitted for App calibrate, waiting on task 793 parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 794 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 795 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash 810b68be655db961829705151bef731c parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246346020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/4daea850-2bc5-4ffe-b3fa-f3db6a07ed83_isr_945869_7.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/4daea850-2bc5-4ffe-b3fa-f3db6a07ed83_isr_945869_7.stderr parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 796 submitted for App characterizeImage, waiting on task 795 parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 796 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 797 submitted for App calibrate, waiting on task 796 parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 797 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 798 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash ec66a6d96e89cec63f5511a3f5cf177c parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246346340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/cb8e6113-79a1-475c-9c12-f33f2cd5ca77_isr_945755_17.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/cb8e6113-79a1-475c-9c12-f33f2cd5ca77_isr_945755_17.stderr parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 799 submitted for App characterizeImage, waiting on task 798 parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 799 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 800 submitted for App calibrate, waiting on task 799 parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 800 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 801 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash 059eb0ff1b12c663d86011af31060e68 parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246346660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a53bb469-ac93-406f-973f-38750e6411ea_isr_945757_47.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a53bb469-ac93-406f-973f-38750e6411ea_isr_945757_47.stderr parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 802 submitted for App characterizeImage, waiting on task 801 parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 802 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 803 submitted for App calibrate, waiting on task 802 parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 803 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 804 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 804 has memoization hash 106a799cca7a1f000a845a5ab28820be parsl.dataflow.memoization INFO: Task 804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246346980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3023b4b3-ac97-4253-b91b-08145e4b1af5_isr_945755_50.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3023b4b3-ac97-4253-b91b-08145e4b1af5_isr_945755_50.stderr parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 805 submitted for App characterizeImage, waiting on task 804 parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 805 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 806 submitted for App calibrate, waiting on task 805 parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 806 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 807 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 807 has memoization hash 1bd86a68c9ae97ac555490e695ba78e6 parsl.dataflow.memoization INFO: Task 807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246346ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/35f66b15-a6f2-4ca1-a39e-f26609a99087_isr_945868_57.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/35f66b15-a6f2-4ca1-a39e-f26609a99087_isr_945868_57.stderr parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 808 submitted for App characterizeImage, waiting on task 807 parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 808 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 809 submitted for App calibrate, waiting on task 808 parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 809 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 810 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 810 has memoization hash 964b7207f6848938a994735c798e85d1 parsl.dataflow.memoization INFO: Task 810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246346fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/23598f03-eb09-4f22-bd65-191e7874b484_isr_945868_14.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/23598f03-eb09-4f22-bd65-191e7874b484_isr_945868_14.stderr parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 811 submitted for App characterizeImage, waiting on task 810 parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 811 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 812 submitted for App calibrate, waiting on task 811 parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 812 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 813 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 813 has memoization hash 710e5c7d323e3d62134f4c1ddb1bc59e parsl.dataflow.memoization INFO: Task 813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463472e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/c098b796-0054-4cb6-9648-60ac9d40c7c5_isr_945756_39.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/c098b796-0054-4cb6-9648-60ac9d40c7c5_isr_945756_39.stderr parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 814 submitted for App characterizeImage, waiting on task 813 parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 814 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 815 submitted for App calibrate, waiting on task 814 parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 815 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 816 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 816 has memoization hash 1302c45c6c75a0f6f60356b97e929a73 parsl.dataflow.memoization INFO: Task 816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/e984bfc5-ad5f-4fb7-ad5c-835040bba7cf_isr_945868_31.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/e984bfc5-ad5f-4fb7-ad5c-835040bba7cf_isr_945868_31.stderr parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 817 submitted for App characterizeImage, waiting on task 816 parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 817 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 818 submitted for App calibrate, waiting on task 817 parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 818 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 819 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 819 has memoization hash b565df00186891eb1403a0a58ca49927 parsl.dataflow.memoization INFO: Task 819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/97f6c2de-1d47-47d6-b5d0-5d73a233c6f2_isr_945756_10.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/97f6c2de-1d47-47d6-b5d0-5d73a233c6f2_isr_945756_10.stderr parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 820 submitted for App characterizeImage, waiting on task 819 parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 820 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 821 submitted for App calibrate, waiting on task 820 parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 821 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 822 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 822 has memoization hash 31c5a23015feb8af9136ec1a57ff9bf3 parsl.dataflow.memoization INFO: Task 822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d3ad9281-63af-4024-8331-5a38c7cb8c81_isr_945868_13.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d3ad9281-63af-4024-8331-5a38c7cb8c81_isr_945868_13.stderr parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 823 submitted for App characterizeImage, waiting on task 822 parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 823 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 824 submitted for App calibrate, waiting on task 823 parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 824 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 825 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 825 has memoization hash 2613893783cc22a635973069167ea092 parsl.dataflow.memoization INFO: Task 825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/2de1d611-3964-4550-be33-249de877734a_isr_945868_46.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/2de1d611-3964-4550-be33-249de877734a_isr_945868_46.stderr parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 826 submitted for App characterizeImage, waiting on task 825 parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 826 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 827 submitted for App calibrate, waiting on task 826 parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 827 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 828 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 828 has memoization hash e44ee332c01c6a921c3619f39f483746 parsl.dataflow.memoization INFO: Task 828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/20f2cfc7-07c8-435b-96de-d6d69219430a_isr_945869_37.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/20f2cfc7-07c8-435b-96de-d6d69219430a_isr_945869_37.stderr parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 829 submitted for App characterizeImage, waiting on task 828 parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 829 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 830 submitted for App calibrate, waiting on task 829 parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 830 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 831 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 831 has memoization hash a045305d746f82a21acc14b2737f5afb parsl.dataflow.memoization INFO: Task 831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/fc11d2d4-8971-4a72-90a4-d0a3d2b85221_isr_945757_18.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/fc11d2d4-8971-4a72-90a4-d0a3d2b85221_isr_945757_18.stderr parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 832 submitted for App characterizeImage, waiting on task 831 parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 832 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 833 submitted for App calibrate, waiting on task 832 parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 833 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 834 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 834 has memoization hash bed32f4702971ee663955f18094fa8d7 parsl.dataflow.memoization INFO: Task 834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/dcba9cf3-7879-4f45-adc2-33c6c07f2b92_isr_945867_62.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/dcba9cf3-7879-4f45-adc2-33c6c07f2b92_isr_945867_62.stderr parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 835 submitted for App characterizeImage, waiting on task 834 parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 835 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 836 submitted for App calibrate, waiting on task 835 parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 836 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 837 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 837 has memoization hash af60e4431dea480e4724dfdf56a142db parsl.dataflow.memoization INFO: Task 837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/01ee1bf3-75e2-49e7-914b-105f2b48c6ba_isr_945868_48.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/01ee1bf3-75e2-49e7-914b-105f2b48c6ba_isr_945868_48.stderr parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 838 submitted for App characterizeImage, waiting on task 837 parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 838 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 839 submitted for App calibrate, waiting on task 838 parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 839 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 840 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 840 has memoization hash d73761d29267f9e4778f4887113a7727 parsl.dataflow.memoization INFO: Task 840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1eb5a2fc-b422-4ca2-8c58-66571a1d28e1_isr_945867_54.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1eb5a2fc-b422-4ca2-8c58-66571a1d28e1_isr_945867_54.stderr parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 841 submitted for App characterizeImage, waiting on task 840 parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 841 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 842 submitted for App calibrate, waiting on task 841 parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 842 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 843 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 843 has memoization hash cfea82a6cf5f9ab404d7ef0c5b2a990f parsl.dataflow.memoization INFO: Task 843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/71632fe1-8200-4813-a47c-5d3d6aee2d44_isr_945756_49.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/71632fe1-8200-4813-a47c-5d3d6aee2d44_isr_945756_49.stderr parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 844 submitted for App characterizeImage, waiting on task 843 parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 844 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 845 submitted for App calibrate, waiting on task 844 parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 845 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 846 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 846 has memoization hash 96eb6c1c6c80984e3b4bd23d0529566b parsl.dataflow.memoization INFO: Task 846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f79b92e5-0d92-43de-90fc-56e87cb196d4_isr_945869_47.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f79b92e5-0d92-43de-90fc-56e87cb196d4_isr_945869_47.stderr parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 847 submitted for App characterizeImage, waiting on task 846 parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 847 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 848 submitted for App calibrate, waiting on task 847 parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 848 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 849 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 849 has memoization hash e8d7c91f8c98a56454dec52f6891be3e parsl.dataflow.memoization INFO: Task 849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/a4a732c0-8aaa-4449-963d-8c4e485b8392_isr_945869_58.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/a4a732c0-8aaa-4449-963d-8c4e485b8392_isr_945869_58.stderr parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 850 submitted for App characterizeImage, waiting on task 849 parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 850 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 851 submitted for App calibrate, waiting on task 850 parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 851 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 852 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 852 has memoization hash 87013e20880a79cf5f06607d59951757 parsl.dataflow.memoization INFO: Task 852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/7aedc253-37c9-47a8-8dc1-2bcaf2a9e0b1_isr_945756_13.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/7aedc253-37c9-47a8-8dc1-2bcaf2a9e0b1_isr_945756_13.stderr parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 853 submitted for App characterizeImage, waiting on task 852 parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 853 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 854 submitted for App calibrate, waiting on task 853 parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 854 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 855 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 855 has memoization hash 9550b820cfd7945895d771f29ed6a3f0 parsl.dataflow.memoization INFO: Task 855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/6228746c-7bdf-4458-a30c-f25ae39eab3c_isr_945755_14.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/6228746c-7bdf-4458-a30c-f25ae39eab3c_isr_945755_14.stderr parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 856 submitted for App characterizeImage, waiting on task 855 parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 856 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 857 submitted for App calibrate, waiting on task 856 parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 857 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 858 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 858 has memoization hash 2afa95bb6c310e6d54fa374f6f584096 parsl.dataflow.memoization INFO: Task 858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/dcf88211-7fa1-4748-b94b-58be40d1c3a2_isr_945755_40.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/dcf88211-7fa1-4748-b94b-58be40d1c3a2_isr_945755_40.stderr parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 859 submitted for App characterizeImage, waiting on task 858 parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 859 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 860 submitted for App calibrate, waiting on task 859 parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 860 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 861 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 861 has memoization hash 0e7ef9505aedde1965082b79f2482a4f parsl.dataflow.memoization INFO: Task 861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/0232bbc6-9a72-40e0-b10a-53baa0e2271d_isr_945867_28.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/0232bbc6-9a72-40e0-b10a-53baa0e2271d_isr_945867_28.stderr parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 862 submitted for App characterizeImage, waiting on task 861 parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 862 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 863 submitted for App calibrate, waiting on task 862 parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 863 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 864 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 864 has memoization hash 5e0786bda7bad2fdc3cc56df2ec81d1a parsl.dataflow.memoization INFO: Task 864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ffaf8af9-637c-4680-a244-c32aa53ef9e2_isr_945868_54.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ffaf8af9-637c-4680-a244-c32aa53ef9e2_isr_945868_54.stderr parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 865 submitted for App characterizeImage, waiting on task 864 parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 865 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 866 submitted for App calibrate, waiting on task 865 parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 866 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 867 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 867 has memoization hash bd8ab94a1a9356aa5d2b8d6ce792871f parsl.dataflow.memoization INFO: Task 867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/05b541c2-e350-42e4-9cb7-a3237df7e701_isr_945869_19.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/05b541c2-e350-42e4-9cb7-a3237df7e701_isr_945869_19.stderr parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 868 submitted for App characterizeImage, waiting on task 867 parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 868 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 869 submitted for App calibrate, waiting on task 868 parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 869 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 870 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 870 has memoization hash ddf6d3d9f8f3ef4e80619faeca5ac4d7 parsl.dataflow.memoization INFO: Task 870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/0f5b7820-32dc-422e-b2a8-25e70919ecb6_isr_945867_26.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/0f5b7820-32dc-422e-b2a8-25e70919ecb6_isr_945867_26.stderr parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 871 submitted for App characterizeImage, waiting on task 870 parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 871 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 872 submitted for App calibrate, waiting on task 871 parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 872 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 873 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 873 has memoization hash a6f97255f52439a0ebb77d7989a0def5 parsl.dataflow.memoization INFO: Task 873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461ab1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b41c7e17-694d-4cf4-9a67-de5e71f8b661_isr_945755_8.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b41c7e17-694d-4cf4-9a67-de5e71f8b661_isr_945755_8.stderr parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 874 submitted for App characterizeImage, waiting on task 873 parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 874 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 875 submitted for App calibrate, waiting on task 874 parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 875 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 876 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 876 has memoization hash 6984164a97c4ff5b78504547db12a5d8 parsl.dataflow.memoization INFO: Task 876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461ab4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/805bf1e4-d876-4ca6-97b6-0013b41490e1_isr_945755_28.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/805bf1e4-d876-4ca6-97b6-0013b41490e1_isr_945755_28.stderr parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 877 submitted for App characterizeImage, waiting on task 876 parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 877 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 878 submitted for App calibrate, waiting on task 877 parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 878 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 879 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 879 has memoization hash e97cfe3802b5fa19dd6760a90c021657 parsl.dataflow.memoization INFO: Task 879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461ab7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/c90c92c4-aa30-4e94-80ff-b8a1e07de967_isr_945869_39.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/c90c92c4-aa30-4e94-80ff-b8a1e07de967_isr_945869_39.stderr parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 880 submitted for App characterizeImage, waiting on task 879 parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 880 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 881 submitted for App calibrate, waiting on task 880 parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 881 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 882 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 882 has memoization hash f176da890936f32240787d885493cc58 parsl.dataflow.memoization INFO: Task 882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461abb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/0a030639-153c-4352-ba9d-92509604fba5_isr_945868_41.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/0a030639-153c-4352-ba9d-92509604fba5_isr_945868_41.stderr parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 883 submitted for App characterizeImage, waiting on task 882 parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 883 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 884 submitted for App calibrate, waiting on task 883 parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 884 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 885 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 885 has memoization hash 57f662af6d9c8d9d2340b147337a8145 parsl.dataflow.memoization INFO: Task 885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461abe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a2130e19-25f9-4802-bfeb-7f4362088e02_isr_945757_23.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a2130e19-25f9-4802-bfeb-7f4362088e02_isr_945757_23.stderr parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 886 submitted for App characterizeImage, waiting on task 885 parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 886 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 887 submitted for App calibrate, waiting on task 886 parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 887 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 888 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 888 has memoization hash 941e18dedb99df0bfc44fb651f0a3d7a parsl.dataflow.memoization INFO: Task 888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c3f946d9-afcb-486e-9504-911f710b32a5_isr_945868_45.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c3f946d9-afcb-486e-9504-911f710b32a5_isr_945868_45.stderr parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 889 submitted for App characterizeImage, waiting on task 888 parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 889 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 890 submitted for App calibrate, waiting on task 889 parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 890 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 891 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 891 has memoization hash 503ce70a961253951d586155bd86187b parsl.dataflow.memoization INFO: Task 891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/5f619edd-ff6e-46d1-9922-2e12e353cfb0_isr_945868_16.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/5f619edd-ff6e-46d1-9922-2e12e353cfb0_isr_945868_16.stderr parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 892 submitted for App characterizeImage, waiting on task 891 parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 892 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 893 submitted for App calibrate, waiting on task 892 parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 893 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 894 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 894 has memoization hash e63a3422118ff58eb2dd51619669def0 parsl.dataflow.memoization INFO: Task 894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/44f719d3-092c-4958-9d23-91a2fe084b82_isr_945756_31.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/44f719d3-092c-4958-9d23-91a2fe084b82_isr_945756_31.stderr parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 895 submitted for App characterizeImage, waiting on task 894 parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 895 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 896 submitted for App calibrate, waiting on task 895 parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 896 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 897 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 897 has memoization hash f05490d65a202596dfa8eb676fbbfbb5 parsl.dataflow.memoization INFO: Task 897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/992d5294-3547-4f25-9b1f-bd507e864d0b_isr_945869_23.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/992d5294-3547-4f25-9b1f-bd507e864d0b_isr_945869_23.stderr parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 898 submitted for App characterizeImage, waiting on task 897 parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 898 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 899 submitted for App calibrate, waiting on task 898 parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 899 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 900 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 900 has memoization hash 32a444da77d7672c0353cf5ae14090bf parsl.dataflow.memoization INFO: Task 900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/e238fd75-56f1-4b74-aa74-908f5df8fa83_isr_945757_10.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/e238fd75-56f1-4b74-aa74-908f5df8fa83_isr_945757_10.stderr parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 901 submitted for App characterizeImage, waiting on task 900 parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 901 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 902 submitted for App calibrate, waiting on task 901 parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 902 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 903 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 903 has memoization hash e7f3922e123b382cca9aa54ce699bce5 parsl.dataflow.memoization INFO: Task 903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/0972054f-d3c5-4662-b345-23ea2e3aeb4f_isr_945757_12.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/0972054f-d3c5-4662-b345-23ea2e3aeb4f_isr_945757_12.stderr parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 904 submitted for App characterizeImage, waiting on task 903 parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 904 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 905 submitted for App calibrate, waiting on task 904 parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 905 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 906 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 906 has memoization hash 2227d010bc92be36d37265b0e85c1789 parsl.dataflow.memoization INFO: Task 906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 18:33:38 proc_lsst.multi:170 [INFO] [multi] job_id 17020629 proc_lsst.multi INFO: [multi] job_id 17020629 2024-03-13 18:33:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 18:33:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020629 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020629 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17020629 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-13 18:33:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-13 18:33:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 18:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b677757e-085e-4c9e-a746-d535d8543945_isr_945755_46.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b677757e-085e-4c9e-a746-d535d8543945_isr_945755_46.stderr parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 907 submitted for App characterizeImage, waiting on task 906 parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 907 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 908 submitted for App calibrate, waiting on task 907 parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 908 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 909 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 909 has memoization hash a4fb1d494af3ae740e312a8fe9b0ea68 parsl.dataflow.memoization INFO: Task 909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/7d32b846-4754-43c6-8eae-ea04c36e4014_isr_945869_2.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/7d32b846-4754-43c6-8eae-ea04c36e4014_isr_945869_2.stderr parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 910 submitted for App characterizeImage, waiting on task 909 parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 910 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 911 submitted for App calibrate, waiting on task 910 parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 911 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 912 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 912 has memoization hash a0779bf9f895ee1490bbc97e7f9564d4 parsl.dataflow.memoization INFO: Task 912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f2a13a97-1718-4e8a-9d22-f0b1fd9fa1da_isr_945869_18.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f2a13a97-1718-4e8a-9d22-f0b1fd9fa1da_isr_945869_18.stderr parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 913 submitted for App characterizeImage, waiting on task 912 parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 913 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 914 submitted for App calibrate, waiting on task 913 parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 914 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 915 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 915 has memoization hash d8663c33e3bf2045d65bc9a358ac3b15 parsl.dataflow.memoization INFO: Task 915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2f492838-739b-41e2-9cdf-062bf67a1f50_isr_945867_23.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2f492838-739b-41e2-9cdf-062bf67a1f50_isr_945867_23.stderr parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 916 submitted for App characterizeImage, waiting on task 915 parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 916 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 917 submitted for App calibrate, waiting on task 916 parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 917 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 918 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 918 has memoization hash 8bc4e4420a5ee0483ca398a6f557344c parsl.dataflow.memoization INFO: Task 918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/2be981d6-5505-48cb-9489-639b4dd4100f_isr_945755_44.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/2be981d6-5505-48cb-9489-639b4dd4100f_isr_945755_44.stderr parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 919 submitted for App characterizeImage, waiting on task 918 parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 919 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 920 submitted for App calibrate, waiting on task 919 parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 920 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 921 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 921 has memoization hash 478cd3a3844a59446b849ac7f8814aed parsl.dataflow.memoization INFO: Task 921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/61f1264b-9fd9-4e16-87dc-828b958f3dc4_isr_945756_3.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/61f1264b-9fd9-4e16-87dc-828b958f3dc4_isr_945756_3.stderr parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 922 submitted for App characterizeImage, waiting on task 921 parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 922 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 923 submitted for App calibrate, waiting on task 922 parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 923 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 924 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 924 has memoization hash 80644e4ff1df35952cd75ace5f380b10 parsl.dataflow.memoization INFO: Task 924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462296c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9602c900-dcd9-407c-a00c-5fa663d2328a_isr_945868_28.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9602c900-dcd9-407c-a00c-5fa663d2328a_isr_945868_28.stderr parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 925 submitted for App characterizeImage, waiting on task 924 parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 925 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 926 submitted for App calibrate, waiting on task 925 parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 926 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 927 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 927 has memoization hash db25bae84f7cc0aee8d9d0815b4caa0d parsl.dataflow.memoization INFO: Task 927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462293a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/60c40a2e-6038-4969-8256-a5a65d1d7332_isr_945757_44.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/60c40a2e-6038-4969-8256-a5a65d1d7332_isr_945757_44.stderr parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 928 submitted for App characterizeImage, waiting on task 927 parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 928 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 929 submitted for App calibrate, waiting on task 928 parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 929 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 930 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 930 has memoization hash 32ef8fba6b716fc62707bc2bb1888159 parsl.dataflow.memoization INFO: Task 930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246229080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/51fbc5f0-f4ae-4cf6-b957-4b79dcb8eecd_isr_945755_13.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/51fbc5f0-f4ae-4cf6-b957-4b79dcb8eecd_isr_945755_13.stderr parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 931 submitted for App characterizeImage, waiting on task 930 parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 931 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 932 submitted for App calibrate, waiting on task 931 parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 932 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 933 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 933 has memoization hash 0761ec6bdd7ccb8df884879c41507a21 parsl.dataflow.memoization INFO: Task 933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246228d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/8bbe791d-ca8b-4f5d-8bd3-86ba22a7f38e_isr_945756_6.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/8bbe791d-ca8b-4f5d-8bd3-86ba22a7f38e_isr_945756_6.stderr parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 934 submitted for App characterizeImage, waiting on task 933 parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 934 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 935 submitted for App calibrate, waiting on task 934 parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 935 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 936 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 936 has memoization hash 21051a311d2a06cd852eb87e15529f87 parsl.dataflow.memoization INFO: Task 936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246228a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/ffec0a83-fe20-419c-a5ba-60defbe5b179_isr_945869_43.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/ffec0a83-fe20-419c-a5ba-60defbe5b179_isr_945869_43.stderr parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 937 submitted for App characterizeImage, waiting on task 936 parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 937 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 938 submitted for App calibrate, waiting on task 937 parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 938 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 939 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 939 has memoization hash 8c583c18c8e3f885cd8962b830db93ae parsl.dataflow.memoization INFO: Task 939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246228720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d5684a72-352d-406d-aa70-1f42cdeedaff_isr_945757_58.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d5684a72-352d-406d-aa70-1f42cdeedaff_isr_945757_58.stderr parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 940 submitted for App characterizeImage, waiting on task 939 parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 940 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 941 submitted for App calibrate, waiting on task 940 parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 941 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 942 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 942 has memoization hash 071a1d29bebdd92c419fb0d900f99a03 parsl.dataflow.memoization INFO: Task 942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246229940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 942 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/ac0f42b7-0c81-422c-9bd2-a5ebd91fa512_isr_945756_51.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/ac0f42b7-0c81-422c-9bd2-a5ebd91fa512_isr_945756_51.stderr parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 943 submitted for App characterizeImage, waiting on task 942 parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 943 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 944 submitted for App calibrate, waiting on task 943 parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 944 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 945 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 945 has memoization hash fcc740ba6ea8611941d12a6f9652eb26 parsl.dataflow.memoization INFO: Task 945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246229c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/3f8058e7-6956-41ca-9a20-63ebd617384e_isr_945756_32.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/3f8058e7-6956-41ca-9a20-63ebd617384e_isr_945756_32.stderr parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 946 submitted for App characterizeImage, waiting on task 945 parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 946 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 947 submitted for App calibrate, waiting on task 946 parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 947 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 948 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 948 has memoization hash 90b84b644afde9061520413fee8a88a2 parsl.dataflow.memoization INFO: Task 948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246229f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/7bb88b26-7aeb-4843-ac42-5006a4fc8c88_isr_945869_21.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/7bb88b26-7aeb-4843-ac42-5006a4fc8c88_isr_945869_21.stderr parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 949 submitted for App characterizeImage, waiting on task 948 parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 949 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 950 submitted for App calibrate, waiting on task 949 parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 950 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 951 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 951 has memoization hash fa3afc141b77fd73a74cef09096f1c88 parsl.dataflow.memoization INFO: Task 951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/48a69363-75c7-4e14-8206-41e02853461e_isr_945755_18.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/48a69363-75c7-4e14-8206-41e02853461e_isr_945755_18.stderr parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 952 submitted for App characterizeImage, waiting on task 951 parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 952 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 953 submitted for App calibrate, waiting on task 952 parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 953 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 954 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 954 has memoization hash 8062151b6a1de6af2fa681f104a6d10f parsl.dataflow.memoization INFO: Task 954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b843959c-4ed7-46ad-83e2-c3af90cd0147_isr_945755_54.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b843959c-4ed7-46ad-83e2-c3af90cd0147_isr_945755_54.stderr parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 955 submitted for App characterizeImage, waiting on task 954 parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 955 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 956 submitted for App calibrate, waiting on task 955 parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 956 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 957 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 957 has memoization hash a9953bf5727e6cf095573a927b73b245 parsl.dataflow.memoization INFO: Task 957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/60f05eed-49ac-4181-b793-45ac9c6782bf_isr_945756_23.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/60f05eed-49ac-4181-b793-45ac9c6782bf_isr_945756_23.stderr parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 958 submitted for App characterizeImage, waiting on task 957 parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 958 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 959 submitted for App calibrate, waiting on task 958 parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 959 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 960 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 960 has memoization hash 05ee0aec8794a7408d7f510cfd5ba012 parsl.dataflow.memoization INFO: Task 960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/185ef91a-067c-4e62-b297-c79867b88bcd_isr_945867_24.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/185ef91a-067c-4e62-b297-c79867b88bcd_isr_945867_24.stderr parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 961 submitted for App characterizeImage, waiting on task 960 parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 961 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 962 submitted for App calibrate, waiting on task 961 parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 962 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 963 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 963 has memoization hash cf35cb6bfa4f6683910a374d18f43855 parsl.dataflow.memoization INFO: Task 963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/5ce13f3a-17f8-4d70-85d7-091a4d85ab51_isr_945757_24.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/5ce13f3a-17f8-4d70-85d7-091a4d85ab51_isr_945757_24.stderr parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 964 submitted for App characterizeImage, waiting on task 963 parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 964 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 965 submitted for App calibrate, waiting on task 964 parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 965 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 966 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 966 has memoization hash d2d58fbbf417b1f38316471fdb76c09b parsl.dataflow.memoization INFO: Task 966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/dbcfa76d-6b4a-4f74-ab98-ec18f0def311_isr_945867_6.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/dbcfa76d-6b4a-4f74-ab98-ec18f0def311_isr_945867_6.stderr parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 967 submitted for App characterizeImage, waiting on task 966 parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 967 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 968 submitted for App calibrate, waiting on task 967 parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 968 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 969 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 969 has memoization hash 86150fa544be0f7ec9d63364e5c58578 parsl.dataflow.memoization INFO: Task 969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/61e3df39-a954-451c-b8d4-8c14e0bc85f3_isr_945756_17.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/61e3df39-a954-451c-b8d4-8c14e0bc85f3_isr_945756_17.stderr parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 970 submitted for App characterizeImage, waiting on task 969 parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 970 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 971 submitted for App calibrate, waiting on task 970 parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 971 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 972 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 972 has memoization hash 73c7482c88f61f81a7248546389adf84 parsl.dataflow.memoization INFO: Task 972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e92f2df4-1232-46bc-8cdb-1c4a679dc0d4_isr_945869_14.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e92f2df4-1232-46bc-8cdb-1c4a679dc0d4_isr_945869_14.stderr parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 973 submitted for App characterizeImage, waiting on task 972 parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 973 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 974 submitted for App calibrate, waiting on task 973 parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 974 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 975 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 975 has memoization hash 3b6e0a1183e8f950376ee9af29872911 parsl.dataflow.memoization INFO: Task 975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/c388f6a3-7af2-4569-bca7-e44694c52376_isr_945869_17.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/c388f6a3-7af2-4569-bca7-e44694c52376_isr_945869_17.stderr parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 976 submitted for App characterizeImage, waiting on task 975 parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 976 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 977 submitted for App calibrate, waiting on task 976 parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 977 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 978 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 978 has memoization hash 310b9ab9fac874add8214a8dd93fda91 parsl.dataflow.memoization INFO: Task 978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/30b58ded-ceb5-43c0-bf36-c90df70140e8_isr_945756_58.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/30b58ded-ceb5-43c0-bf36-c90df70140e8_isr_945756_58.stderr parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 979 submitted for App characterizeImage, waiting on task 978 parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 979 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 980 submitted for App calibrate, waiting on task 979 parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 980 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 981 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 981 has memoization hash b7a6bb29fcb3b315eb9800788c2761e9 parsl.dataflow.memoization INFO: Task 981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e93ebc27-4720-4947-b56c-d87f83993ca4_isr_945869_38.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e93ebc27-4720-4947-b56c-d87f83993ca4_isr_945869_38.stderr parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 982 submitted for App characterizeImage, waiting on task 981 parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 982 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 983 submitted for App calibrate, waiting on task 982 parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 983 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 984 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 984 has memoization hash a073ee7039b5f6b4fafe7e4acd20804e parsl.dataflow.memoization INFO: Task 984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246228040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/e41313bd-69be-4a06-a2ce-f477e7ccc6ad_isr_945757_28.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/e41313bd-69be-4a06-a2ce-f477e7ccc6ad_isr_945757_28.stderr parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 985 submitted for App characterizeImage, waiting on task 984 parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 985 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 986 submitted for App calibrate, waiting on task 985 parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 986 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 987 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 987 has memoization hash 8ead752da727076941bc74c690efe2fd parsl.dataflow.memoization INFO: Task 987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/0651664d-ca62-4342-a0df-6f7bc23b2d07_isr_945755_48.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/0651664d-ca62-4342-a0df-6f7bc23b2d07_isr_945755_48.stderr parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 988 submitted for App characterizeImage, waiting on task 987 parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 988 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 989 submitted for App calibrate, waiting on task 988 parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 989 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 990 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 990 has memoization hash d2df74aeb3c8caa28e252225eceaccfc parsl.dataflow.memoization INFO: Task 990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/557949c7-f5f5-49da-939f-f23e5c5c9039_isr_945757_25.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/557949c7-f5f5-49da-939f-f23e5c5c9039_isr_945757_25.stderr parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 991 submitted for App characterizeImage, waiting on task 990 parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 991 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 992 submitted for App calibrate, waiting on task 991 parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 992 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 993 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 993 has memoization hash 442665055de540c57a082457a7bc09c4 parsl.dataflow.memoization INFO: Task 993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e969c954-6cbd-4efc-befa-03726a0daf1f_isr_945755_51.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e969c954-6cbd-4efc-befa-03726a0daf1f_isr_945755_51.stderr parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 994 submitted for App characterizeImage, waiting on task 993 parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 994 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 995 submitted for App calibrate, waiting on task 994 parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 995 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 996 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 996 has memoization hash eb89f9b649b93aa8b17cbc98a82e63d1 parsl.dataflow.memoization INFO: Task 996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a0c8b2e8-f660-4aa9-af2d-11a4ea1b41cb_isr_945757_60.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a0c8b2e8-f660-4aa9-af2d-11a4ea1b41cb_isr_945757_60.stderr parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 997 submitted for App characterizeImage, waiting on task 996 parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 997 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 998 submitted for App calibrate, waiting on task 997 parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 998 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 999 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 999 has memoization hash c3239628755f06bcb82ee5fbc62f054f parsl.dataflow.memoization INFO: Task 999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/1a2961e1-257d-4e03-b212-de2c8d8c8f21_isr_945755_33.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/1a2961e1-257d-4e03-b212-de2c8d8c8f21_isr_945755_33.stderr parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1000 submitted for App characterizeImage, waiting on task 999 parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1000 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1001 submitted for App calibrate, waiting on task 1000 parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1001 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1002 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash e9b92202117138a22a933bc83c907b5c parsl.dataflow.memoization INFO: Task 1002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2dd1616f-439c-4a8c-820d-148a470e52d6_isr_945756_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2dd1616f-439c-4a8c-820d-148a470e52d6_isr_945756_20.stderr parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1003 submitted for App characterizeImage, waiting on task 1002 parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1003 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1004 submitted for App calibrate, waiting on task 1003 parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1004 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1005 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash 263664773e263b50dc0b0b172227e347 parsl.dataflow.memoization INFO: Task 1005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/922a3708-c88d-40da-9a1c-e12cf1ff9052_isr_945869_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/922a3708-c88d-40da-9a1c-e12cf1ff9052_isr_945869_59.stderr parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1006 submitted for App characterizeImage, waiting on task 1005 parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1006 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1007 submitted for App calibrate, waiting on task 1006 parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1007 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1008 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash b46c3d91b2a2e3e6364698769019cf02 parsl.dataflow.memoization INFO: Task 1008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/d362da4e-9cac-4e75-948e-f5b65028536d_isr_945756_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/d362da4e-9cac-4e75-948e-f5b65028536d_isr_945756_25.stderr parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1009 submitted for App characterizeImage, waiting on task 1008 parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1009 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1010 submitted for App calibrate, waiting on task 1009 parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1010 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1011 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1011 has memoization hash 08fd5dc91dd33bd84e4ae4fd01c9a8d8 parsl.dataflow.memoization INFO: Task 1011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246088220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1011 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/31ce2812-944b-4704-bdb4-4600e1294722_isr_945867_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/31ce2812-944b-4704-bdb4-4600e1294722_isr_945867_19.stderr parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1012 submitted for App characterizeImage, waiting on task 1011 parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1012 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1013 submitted for App calibrate, waiting on task 1012 parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1013 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1014 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash 444a5147c05bd67eb08672fa62a9c889 parsl.dataflow.memoization INFO: Task 1014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246088540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/a3c0a83e-640a-4e4b-a56a-ce080ba3cccf_isr_945756_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/a3c0a83e-640a-4e4b-a56a-ce080ba3cccf_isr_945756_27.stderr parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1015 submitted for App characterizeImage, waiting on task 1014 parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1015 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1016 submitted for App calibrate, waiting on task 1015 parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1016 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1017 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash d9331b7b8d0b66809a6f39447896aac5 parsl.dataflow.memoization INFO: Task 1017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246088860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/19c80da6-eb3e-402e-901d-4c1d3ae18dde_isr_945867_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/19c80da6-eb3e-402e-901d-4c1d3ae18dde_isr_945867_35.stderr parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1018 submitted for App characterizeImage, waiting on task 1017 parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1018 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1019 submitted for App calibrate, waiting on task 1018 parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1019 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1020 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash e295115ae332f3853be5f61e3d2f7129 parsl.dataflow.memoization INFO: Task 1020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246088b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/98aff5f1-455c-4267-b6c4-91f55a0eebfc_isr_945868_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/98aff5f1-455c-4267-b6c4-91f55a0eebfc_isr_945868_39.stderr parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1021 submitted for App characterizeImage, waiting on task 1020 parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1021 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1022 submitted for App calibrate, waiting on task 1021 parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1022 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1023 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash c38659664488e96a7db0a82970e4b7f0 parsl.dataflow.memoization INFO: Task 1023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246088ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/dae71855-2081-4e23-8dad-5c91abe22e0d_isr_945756_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/dae71855-2081-4e23-8dad-5c91abe22e0d_isr_945756_45.stderr parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1024 submitted for App characterizeImage, waiting on task 1023 parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1024 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1025 submitted for App calibrate, waiting on task 1024 parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1025 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1026 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash e16079cc2d55c47b931e0fa918643267 parsl.dataflow.memoization INFO: Task 1026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460891c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/c53b9597-9b1c-41f1-b8f3-9c2b414a7faa_isr_945755_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/c53b9597-9b1c-41f1-b8f3-9c2b414a7faa_isr_945755_16.stderr parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1027 submitted for App characterizeImage, waiting on task 1026 parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1027 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1028 submitted for App calibrate, waiting on task 1027 parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1028 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1029 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash 57c96b46e607b65227e1db6320df2828 parsl.dataflow.memoization INFO: Task 1029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460894e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/4a6ad8dc-1cb3-4e9d-bc8d-57deffe61a6a_isr_945757_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/4a6ad8dc-1cb3-4e9d-bc8d-57deffe61a6a_isr_945757_16.stderr parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1030 submitted for App characterizeImage, waiting on task 1029 parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1030 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1031 submitted for App calibrate, waiting on task 1030 parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1031 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1032 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash 67f3a2ff71fffe9572ac875c89aaccab parsl.dataflow.memoization INFO: Task 1032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246089800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/4d5b8aa0-5056-458c-a065-cd1c4f0a5afd_isr_945757_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/4d5b8aa0-5056-458c-a065-cd1c4f0a5afd_isr_945757_13.stderr parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1033 submitted for App characterizeImage, waiting on task 1032 parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1033 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1034 submitted for App calibrate, waiting on task 1033 parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1034 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1035 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash 4da60e7a658b2fbb2054fe9c6feeb32e parsl.dataflow.memoization INFO: Task 1035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246089b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c2a962f7-307d-47cb-9582-e6280183b0ea_isr_945868_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c2a962f7-307d-47cb-9582-e6280183b0ea_isr_945868_42.stderr parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1036 submitted for App characterizeImage, waiting on task 1035 parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1036 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1037 submitted for App calibrate, waiting on task 1036 parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1037 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1038 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash 972eb685525ce8bd72239dd912fd916c parsl.dataflow.memoization INFO: Task 1038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246089e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/ace99fde-37ab-490c-912e-9696217a36f1_isr_945867_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/ace99fde-37ab-490c-912e-9696217a36f1_isr_945867_52.stderr parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1039 submitted for App characterizeImage, waiting on task 1038 parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1039 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1040 submitted for App calibrate, waiting on task 1039 parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1040 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1041 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash ddaeed9a1d28569f39a6348910064ea6 parsl.dataflow.memoization INFO: Task 1041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c0a6d980-cb42-4512-a08d-7ab3f2a606c2_isr_945868_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c0a6d980-cb42-4512-a08d-7ab3f2a606c2_isr_945868_30.stderr parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1042 submitted for App characterizeImage, waiting on task 1041 parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1042 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1043 submitted for App calibrate, waiting on task 1042 parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1043 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1044 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash e83835986358a5764de7380d38cff0b0 parsl.dataflow.memoization INFO: Task 1044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/efe70310-0635-4d4d-b5f0-66520b2ca4e2_isr_945756_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/efe70310-0635-4d4d-b5f0-66520b2ca4e2_isr_945756_16.stderr parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1045 submitted for App characterizeImage, waiting on task 1044 parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1045 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1046 submitted for App calibrate, waiting on task 1045 parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1046 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1047 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash 23e1cf3abed6a853d22bb1b8fad98837 parsl.dataflow.memoization INFO: Task 1047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/e3b3e3b4-97ef-41e8-aaa9-137beae41810_isr_945756_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/e3b3e3b4-97ef-41e8-aaa9-137beae41810_isr_945756_4.stderr parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1048 submitted for App characterizeImage, waiting on task 1047 parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1048 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1049 submitted for App calibrate, waiting on task 1048 parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1049 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1050 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash eba1b9633ce358b479aee923845901d8 parsl.dataflow.memoization INFO: Task 1050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ed802df7-7549-4a4b-9d80-6b353b5accfb_isr_945868_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ed802df7-7549-4a4b-9d80-6b353b5accfb_isr_945868_55.stderr parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1051 submitted for App characterizeImage, waiting on task 1050 parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1051 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1052 submitted for App calibrate, waiting on task 1051 parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1052 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1053 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash 12e47533c66c3b10e766368c8709ac33 parsl.dataflow.memoization INFO: Task 1053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e8285f40-0fb5-44f0-8c98-d091ead19fd6_isr_945755_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e8285f40-0fb5-44f0-8c98-d091ead19fd6_isr_945755_30.stderr parsl.dataflow.dflow DEBUG: Task 1054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1054 submitted for App characterizeImage, waiting on task 1053 parsl.dataflow.dflow DEBUG: Task 1054 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1054 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1055 submitted for App calibrate, waiting on task 1054 parsl.dataflow.dflow DEBUG: Task 1055 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1055 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1056 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1056 has memoization hash f2b7b2b63520688bf603ab6884767e0d parsl.dataflow.memoization INFO: Task 1056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1056 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8dd92384-db5e-4192-8304-70055d7702e5_isr_945755_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8dd92384-db5e-4192-8304-70055d7702e5_isr_945755_47.stderr parsl.dataflow.dflow DEBUG: Task 1057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1057 submitted for App characterizeImage, waiting on task 1056 parsl.dataflow.dflow DEBUG: Task 1057 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1057 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1058 submitted for App calibrate, waiting on task 1057 parsl.dataflow.dflow DEBUG: Task 1058 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1058 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1059 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1059 has memoization hash 0387f513110c03badc345ffaa61cbc39 parsl.dataflow.memoization INFO: Task 1059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1059 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8e2f3c72-d0d2-43b5-98f0-c7a5230341ce_isr_945867_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8e2f3c72-d0d2-43b5-98f0-c7a5230341ce_isr_945867_55.stderr parsl.dataflow.dflow DEBUG: Task 1060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1060 submitted for App characterizeImage, waiting on task 1059 parsl.dataflow.dflow DEBUG: Task 1060 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1060 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1061 submitted for App calibrate, waiting on task 1060 parsl.dataflow.dflow DEBUG: Task 1061 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1061 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1062 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1062 has memoization hash e250870bfaf6c4d9aa81572ad828b3cb parsl.dataflow.memoization INFO: Task 1062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1062 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/5cf332fb-e191-4cf0-88a0-198598412807_isr_945869_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/5cf332fb-e191-4cf0-88a0-198598412807_isr_945869_42.stderr parsl.dataflow.dflow DEBUG: Task 1063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1063 submitted for App characterizeImage, waiting on task 1062 parsl.dataflow.dflow DEBUG: Task 1063 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1063 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1064 submitted for App calibrate, waiting on task 1063 parsl.dataflow.dflow DEBUG: Task 1064 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1064 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1065 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1065 has memoization hash bccc1b119a9a558b238f60207295891e parsl.dataflow.memoization INFO: Task 1065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1065 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/9db927fb-8650-4039-a61d-f7b6ef368daf_isr_945757_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/9db927fb-8650-4039-a61d-f7b6ef368daf_isr_945757_52.stderr parsl.dataflow.dflow DEBUG: Task 1066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1066 submitted for App characterizeImage, waiting on task 1065 parsl.dataflow.dflow DEBUG: Task 1066 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1066 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1067 submitted for App calibrate, waiting on task 1066 parsl.dataflow.dflow DEBUG: Task 1067 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1067 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1068 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1068 has memoization hash 5608bde9512c301dfe0e5a200ce852d7 parsl.dataflow.memoization INFO: Task 1068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1068 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/97e84b8e-cbea-4728-9263-d46a66e7163b_isr_945868_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/97e84b8e-cbea-4728-9263-d46a66e7163b_isr_945868_59.stderr parsl.dataflow.dflow DEBUG: Task 1069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1069 submitted for App characterizeImage, waiting on task 1068 parsl.dataflow.dflow DEBUG: Task 1069 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1069 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1070 submitted for App calibrate, waiting on task 1069 parsl.dataflow.dflow DEBUG: Task 1070 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1070 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1071 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1071 has memoization hash 4e2d83c5794fd367ef8e716c4e9317a0 parsl.dataflow.memoization INFO: Task 1071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1071 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/aeb10ca6-6258-467f-8ab5-92d74cb3cb88_isr_945867_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/aeb10ca6-6258-467f-8ab5-92d74cb3cb88_isr_945867_4.stderr parsl.dataflow.dflow DEBUG: Task 1072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1072 submitted for App characterizeImage, waiting on task 1071 parsl.dataflow.dflow DEBUG: Task 1072 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1072 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1073 submitted for App calibrate, waiting on task 1072 parsl.dataflow.dflow DEBUG: Task 1073 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1073 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1074 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1074 has memoization hash 5bef5395e5111c9bacc0d56a2bf8544f parsl.dataflow.memoization INFO: Task 1074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1074 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8bdc49a1-783b-49cc-8a94-d880bb890129_isr_945869_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8bdc49a1-783b-49cc-8a94-d880bb890129_isr_945869_61.stderr parsl.dataflow.dflow DEBUG: Task 1075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1075 submitted for App characterizeImage, waiting on task 1074 parsl.dataflow.dflow DEBUG: Task 1075 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1075 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1076 submitted for App calibrate, waiting on task 1075 parsl.dataflow.dflow DEBUG: Task 1076 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1076 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1077 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1077 has memoization hash 6f8d6f4c47f37cdaa99d21b1eaad090b parsl.dataflow.memoization INFO: Task 1077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1077 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/6ccf2c1a-bd61-42e4-a61f-df86db50f873_isr_945867_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/6ccf2c1a-bd61-42e4-a61f-df86db50f873_isr_945867_51.stderr parsl.dataflow.dflow DEBUG: Task 1078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1078 submitted for App characterizeImage, waiting on task 1077 parsl.dataflow.dflow DEBUG: Task 1078 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1078 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1079 submitted for App calibrate, waiting on task 1078 parsl.dataflow.dflow DEBUG: Task 1079 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1079 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1080 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1080 has memoization hash 8b67205d7c0df8e89ccd58fdd890d49d parsl.dataflow.memoization INFO: Task 1080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1080 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/e589c49c-4591-4689-aec0-8529d461232d_isr_945868_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/e589c49c-4591-4689-aec0-8529d461232d_isr_945868_9.stderr parsl.dataflow.dflow DEBUG: Task 1081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1081 submitted for App characterizeImage, waiting on task 1080 parsl.dataflow.dflow DEBUG: Task 1081 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1081 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1082 submitted for App calibrate, waiting on task 1081 parsl.dataflow.dflow DEBUG: Task 1082 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1082 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1083 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1083 has memoization hash 4d0c1fdc340fa8b86fd031b956b70d66 parsl.dataflow.memoization INFO: Task 1083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1083 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/0148b963-3674-482f-8aed-4eb2d680a63d_isr_945757_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/0148b963-3674-482f-8aed-4eb2d680a63d_isr_945757_30.stderr parsl.dataflow.dflow DEBUG: Task 1084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1084 submitted for App characterizeImage, waiting on task 1083 parsl.dataflow.dflow DEBUG: Task 1084 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1084 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1085 submitted for App calibrate, waiting on task 1084 parsl.dataflow.dflow DEBUG: Task 1085 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1085 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1086 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1086 has memoization hash 5f44f3d1d353075490aaacdcfc2827e4 parsl.dataflow.memoization INFO: Task 1086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1086 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/dd002729-4ef2-4369-a1e1-eb84677ccc36_isr_945756_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/dd002729-4ef2-4369-a1e1-eb84677ccc36_isr_945756_47.stderr parsl.dataflow.dflow DEBUG: Task 1087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1087 submitted for App characterizeImage, waiting on task 1086 parsl.dataflow.dflow DEBUG: Task 1087 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1087 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1088 submitted for App calibrate, waiting on task 1087 parsl.dataflow.dflow DEBUG: Task 1088 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1088 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1089 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1089 has memoization hash 5091682e979abdf310e486a56af7902c parsl.dataflow.memoization INFO: Task 1089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 18:33:38 proc_lsst.multi:170 [INFO] [multi] job_id 17020630 proc_lsst.multi INFO: [multi] job_id 17020630 2024-03-13 18:33:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020630 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020630 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17020630 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-13 18:33:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-13 18:33:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1089 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e63bef9f-1bed-4de7-98f5-7f311ce81047_isr_945755_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e63bef9f-1bed-4de7-98f5-7f311ce81047_isr_945755_31.stderr parsl.dataflow.dflow DEBUG: Task 1090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1090 submitted for App characterizeImage, waiting on task 1089 parsl.dataflow.dflow DEBUG: Task 1090 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1090 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1091 submitted for App calibrate, waiting on task 1090 parsl.dataflow.dflow DEBUG: Task 1091 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1091 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1092 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1092 has memoization hash 63cb4a923f207bb0020506ebdbcd037a parsl.dataflow.memoization INFO: Task 1092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1092 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/6775f9d3-08f0-4a94-b523-d66058378b00_isr_945757_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/6775f9d3-08f0-4a94-b523-d66058378b00_isr_945757_4.stderr parsl.dataflow.dflow DEBUG: Task 1093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1093 submitted for App characterizeImage, waiting on task 1092 parsl.dataflow.dflow DEBUG: Task 1093 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1093 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1094 submitted for App calibrate, waiting on task 1093 parsl.dataflow.dflow DEBUG: Task 1094 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1094 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1095 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1095 has memoization hash e212f6ab7c79886bd348ca803fbcb7c6 parsl.dataflow.memoization INFO: Task 1095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1095 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/c12dbaf2-b061-4020-8399-7415662b6e9f_isr_945867_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/c12dbaf2-b061-4020-8399-7415662b6e9f_isr_945867_3.stderr parsl.dataflow.dflow DEBUG: Task 1096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1096 submitted for App characterizeImage, waiting on task 1095 parsl.dataflow.dflow DEBUG: Task 1096 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1096 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1097 submitted for App calibrate, waiting on task 1096 parsl.dataflow.dflow DEBUG: Task 1097 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1097 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1098 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1098 has memoization hash 3ca88035720fd2b222776fe8d09415e7 parsl.dataflow.memoization INFO: Task 1098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1098 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/527557a9-b9a4-4332-b751-523e2f4e962b_isr_945757_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/527557a9-b9a4-4332-b751-523e2f4e962b_isr_945757_53.stderr parsl.dataflow.dflow DEBUG: Task 1099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1099 submitted for App characterizeImage, waiting on task 1098 parsl.dataflow.dflow DEBUG: Task 1099 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1099 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1100 submitted for App calibrate, waiting on task 1099 parsl.dataflow.dflow DEBUG: Task 1100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1101 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1101 has memoization hash 0f65339178332198ac6c79df315746f3 parsl.dataflow.memoization INFO: Task 1101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1101 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/f827be57-aa3c-49fc-b3a9-b1577262d7b6_isr_945756_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/f827be57-aa3c-49fc-b3a9-b1577262d7b6_isr_945756_28.stderr parsl.dataflow.dflow DEBUG: Task 1102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1102 submitted for App characterizeImage, waiting on task 1101 parsl.dataflow.dflow DEBUG: Task 1102 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1102 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1103 submitted for App calibrate, waiting on task 1102 parsl.dataflow.dflow DEBUG: Task 1103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1104 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1104 has memoization hash e46d04753473156441e7f6f3267e6483 parsl.dataflow.memoization INFO: Task 1104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460be0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1104 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/263a7e79-0ca8-4aa5-996f-0036ead6faa7_isr_945867_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/263a7e79-0ca8-4aa5-996f-0036ead6faa7_isr_945867_61.stderr parsl.dataflow.dflow DEBUG: Task 1105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1105 submitted for App characterizeImage, waiting on task 1104 parsl.dataflow.dflow DEBUG: Task 1105 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1105 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1106 submitted for App calibrate, waiting on task 1105 parsl.dataflow.dflow DEBUG: Task 1106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1107 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1107 has memoization hash d1e5e7360889a791c2f6aec35ee24690 parsl.dataflow.memoization INFO: Task 1107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460be480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1107 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/7e89b5b9-df3a-438c-b026-df5d10898b60_isr_945757_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/7e89b5b9-df3a-438c-b026-df5d10898b60_isr_945757_2.stderr parsl.dataflow.dflow DEBUG: Task 1108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1108 submitted for App characterizeImage, waiting on task 1107 parsl.dataflow.dflow DEBUG: Task 1108 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1108 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1109 submitted for App calibrate, waiting on task 1108 parsl.dataflow.dflow DEBUG: Task 1109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1110 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1110 has memoization hash 96e2758d965d4ece6e9684fdbd90e6e0 parsl.dataflow.memoization INFO: Task 1110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924610dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1110 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/49312e09-1884-47a8-8b7c-7986c1e33a90_isr_945755_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/49312e09-1884-47a8-8b7c-7986c1e33a90_isr_945755_11.stderr parsl.dataflow.dflow DEBUG: Task 1111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1111 submitted for App characterizeImage, waiting on task 1110 parsl.dataflow.dflow DEBUG: Task 1111 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1111 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1112 submitted for App calibrate, waiting on task 1111 parsl.dataflow.dflow DEBUG: Task 1112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1113 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1113 has memoization hash d4990d27d526e7fcc9c9a96cea686f63 parsl.dataflow.memoization INFO: Task 1113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924610d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1113 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4fb30e2e-343e-477b-b1b5-1fd00d153bfc_isr_945867_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4fb30e2e-343e-477b-b1b5-1fd00d153bfc_isr_945867_50.stderr parsl.dataflow.dflow DEBUG: Task 1114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1114 submitted for App characterizeImage, waiting on task 1113 parsl.dataflow.dflow DEBUG: Task 1114 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1114 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1115 submitted for App calibrate, waiting on task 1114 parsl.dataflow.dflow DEBUG: Task 1115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1115 has outstanding dependencies, so launch_if_ready skipping 2024-03-13 18:33:39 proc_lsst.multi:170 [INFO] [multi] job_id 17020631 proc_lsst.multi INFO: [multi] job_id 17020631 2024-03-13 18:33:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 18:33:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:39 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020631 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020631 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17020631 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-13 18:33:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-13 18:33:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 18:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:39 proc_lsst.multi:170 [INFO] [multi] job_id 17020632 proc_lsst.multi INFO: [multi] job_id 17020632 2024-03-13 18:33:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 18:33:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:39 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020632 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020632 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17020632 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-13 18:33:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-13 18:33:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 18:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:40 proc_lsst.multi:170 [INFO] [multi] job_id 17020633 proc_lsst.multi INFO: [multi] job_id 17020633 2024-03-13 18:33:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 18:33:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020633 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020633 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17020633 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-13 18:33:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-13 18:33:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 18:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:40 proc_lsst.multi:170 [INFO] [multi] job_id 17020634 proc_lsst.multi INFO: [multi] job_id 17020634 2024-03-13 18:33:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 18:33:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020634 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020634 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17020634 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-13 18:33:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-13 18:33:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 18:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:40 proc_lsst.multi:170 [INFO] [multi] job_id 17020635 proc_lsst.multi INFO: [multi] job_id 17020635 2024-03-13 18:33:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 18:33:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020635 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020635 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17020635 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-13 18:33:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-13 18:33:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 18:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:41 proc_lsst.multi:170 [INFO] [multi] job_id 17020636 proc_lsst.multi INFO: [multi] job_id 17020636 2024-03-13 18:33:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 18:33:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020636 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020636 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17020636 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-13 18:33:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-13 18:33:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 18:33:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:41 proc_lsst.multi:170 [INFO] [multi] job_id 17020637 proc_lsst.multi INFO: [multi] job_id 17020637 2024-03-13 18:33:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 18:33:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020637 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020637 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17020637 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-13 18:33:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-13 18:33:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 18:33:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:42 proc_lsst.multi:170 [INFO] [multi] job_id 17020638 proc_lsst.multi INFO: [multi] job_id 17020638 2024-03-13 18:33:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 18:33:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020638 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020638 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17020638 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-13 18:33:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-13 18:33:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 18:33:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:42 proc_lsst.multi:170 [INFO] [multi] job_id 17020639 proc_lsst.multi INFO: [multi] job_id 17020639 2024-03-13 18:33:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 18:33:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020639 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020639 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17020639 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-13 18:33:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-13 18:33:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 18:33:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:43 proc_lsst.multi:170 [INFO] [multi] job_id 17020640 proc_lsst.multi INFO: [multi] job_id 17020640 2024-03-13 18:33:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 18:33:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020640 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020640 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17020640 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-13 18:33:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-13 18:33:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 18:33:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:43 proc_lsst.multi:170 [INFO] [multi] job_id 17020641 proc_lsst.multi INFO: [multi] job_id 17020641 2024-03-13 18:33:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 18:33:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020641 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020641 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17020641 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-13 18:33:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-13 18:33:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 18:33:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:43 proc_lsst.multi:170 [INFO] [multi] job_id 17020642 proc_lsst.multi INFO: [multi] job_id 17020642 2024-03-13 18:33:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 18:33:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020642 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020642 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17020642 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-13 18:33:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-13 18:33:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 18:33:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:43 proc_lsst.multi:170 [INFO] [multi] job_id 17020643 proc_lsst.multi INFO: [multi] job_id 17020643 2024-03-13 18:33:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 18:33:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020643 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020643 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17020643 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-13 18:33:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-13 18:33:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 18:33:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:44 proc_lsst.multi:170 [INFO] [multi] job_id 17020644 proc_lsst.multi INFO: [multi] job_id 17020644 2024-03-13 18:33:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 18:33:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020644 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020644 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17020644 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-13 18:33:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-13 18:33:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 18:33:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:44 proc_lsst.multi:170 [INFO] [multi] job_id 17020645 proc_lsst.multi INFO: [multi] job_id 17020645 2024-03-13 18:33:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:33:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020645 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020645 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17020645 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-13 18:33:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-13 18:33:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:33:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:44 proc_lsst.multi:170 [INFO] [multi] job_id 17020646 proc_lsst.multi INFO: [multi] job_id 17020646 2024-03-13 18:33:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 18:33:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020646 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020646 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17020646 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-13 18:33:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-13 18:33:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 18:33:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:45 proc_lsst.multi:170 [INFO] [multi] job_id 17020647 proc_lsst.multi INFO: [multi] job_id 17020647 2024-03-13 18:33:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 18:33:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020647 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020647 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17020647 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-13 18:33:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-13 18:33:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 18:33:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:45 proc_lsst.multi:170 [INFO] [multi] job_id 17020648 proc_lsst.multi INFO: [multi] job_id 17020648 2024-03-13 18:33:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 18:33:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020648 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020648 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17020648 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-13 18:33:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-13 18:33:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 18:33:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:45 proc_lsst.multi:170 [INFO] [multi] job_id 17020649 proc_lsst.multi INFO: [multi] job_id 17020649 2024-03-13 18:33:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 18:33:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020649 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020649 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17020649 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-13 18:33:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-13 18:33:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 18:33:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:46 proc_lsst.multi:170 [INFO] [multi] job_id 17020650 proc_lsst.multi INFO: [multi] job_id 17020650 2024-03-13 18:33:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 18:33:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020650 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020650 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17020650 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-13 18:33:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-13 18:33:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 18:33:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:46 proc_lsst.multi:170 [INFO] [multi] job_id 17020651 proc_lsst.multi INFO: [multi] job_id 17020651 2024-03-13 18:33:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 18:33:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020651 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020651 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17020651 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-13 18:33:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-13 18:33:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 18:33:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:46 proc_lsst.multi:170 [INFO] [multi] job_id 17020652 proc_lsst.multi INFO: [multi] job_id 17020652 2024-03-13 18:33:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 18:33:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020652 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020652 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17020652 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-13 18:33:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-13 18:33:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 18:33:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:47 proc_lsst.multi:170 [INFO] [multi] job_id 17020653 proc_lsst.multi INFO: [multi] job_id 17020653 2024-03-13 18:33:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 18:33:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020653 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020653 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17020653 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-13 18:33:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-13 18:33:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 18:33:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:47 proc_lsst.multi:170 [INFO] [multi] job_id 17020654 proc_lsst.multi INFO: [multi] job_id 17020654 2024-03-13 18:33:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 18:33:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020654 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020654 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17020654 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-13 18:33:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-13 18:33:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 18:33:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:47 proc_lsst.multi:170 [INFO] [multi] job_id 17020655 proc_lsst.multi INFO: [multi] job_id 17020655 2024-03-13 18:33:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17020655 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17020655 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17020655 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-13 18:33:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-13 18:33:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 18:33:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:48 proc_lsst.multi:170 [INFO] [multi] job_id 17020656 proc_lsst.multi INFO: [multi] job_id 17020656 2024-03-13 18:33:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:48 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17020656 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17020656 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17020656 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-13 18:33:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-13 18:33:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:48 proc_lsst.multi:170 [INFO] [multi] job_id 17020657 proc_lsst.multi INFO: [multi] job_id 17020657 2024-03-13 18:33:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 18:33:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:48 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17020657 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17020657 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17020657 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 33, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/32 running/pending blocks, and 16 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 33 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 60 more blocks parsl.executors.status_handling INFO: Scaling out by 60 blocks parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-13 18:33:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-13 18:33:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 18:33:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:49 proc_lsst.multi:170 [INFO] [multi] job_id 17020658 proc_lsst.multi INFO: [multi] job_id 17020658 2024-03-13 18:33:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 18:33:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:49 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17020658 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17020658 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17020658 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-13 18:33:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-13 18:33:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 18:33:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:49 proc_lsst.multi:170 [INFO] [multi] job_id 17020659 proc_lsst.multi INFO: [multi] job_id 17020659 2024-03-13 18:33:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 18:33:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:49 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17020659 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17020659 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17020659 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-13 18:33:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-13 18:33:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 18:33:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:50 proc_lsst.multi:170 [INFO] [multi] job_id 17020660 proc_lsst.multi INFO: [multi] job_id 17020660 2024-03-13 18:33:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:50 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17020660 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17020660 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17020660 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-13 18:33:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-13 18:33:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 18:33:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:50 proc_lsst.multi:170 [INFO] [multi] job_id 17020661 proc_lsst.multi INFO: [multi] job_id 17020661 2024-03-13 18:33:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020661 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020661 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17020661 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-13 18:33:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-13 18:33:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:51 proc_lsst.multi:170 [INFO] [multi] job_id 17020662 proc_lsst.multi INFO: [multi] job_id 17020662 2024-03-13 18:33:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 18:33:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020662 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020662 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17020662 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-13 18:33:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-13 18:33:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 18:33:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:51 proc_lsst.multi:170 [INFO] [multi] job_id 17020663 proc_lsst.multi INFO: [multi] job_id 17020663 2024-03-13 18:33:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 18:33:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020663 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020663 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17020663 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-13 18:33:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-13 18:33:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 18:33:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:52 proc_lsst.multi:170 [INFO] [multi] job_id 17020664 proc_lsst.multi INFO: [multi] job_id 17020664 2024-03-13 18:33:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 18:33:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020664 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020664 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17020664 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-13 18:33:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-13 18:33:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 18:33:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:52 proc_lsst.multi:170 [INFO] [multi] job_id 17020665 proc_lsst.multi INFO: [multi] job_id 17020665 2024-03-13 18:33:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020665 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020665 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17020665 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-13 18:33:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-13 18:33:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0eee71cabc609a56c552a7e7510f0a0e with result from task 3 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash f6a035983b64891ab1ef02d6a1fbfb7a parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247915d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/4c493b7f-5700-40a7-85e3-98bdd8a5988e_characterizeImage_945868_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/4c493b7f-5700-40a7-85e3-98bdd8a5988e_characterizeImage_945868_11.stderr parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d0180f42-548b-4c95-a348-681b7ecc1b21_isr_945868_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d0180f42-548b-4c95-a348-681b7ecc1b21_isr_945868_11.stderr 2024-03-13 18:33:53 proc_lsst.multi:170 [INFO] [multi] job_id 17020666 proc_lsst.multi INFO: [multi] job_id 17020666 2024-03-13 18:33:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 18:33:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:53 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020666 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020666 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17020666 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-13 18:33:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-13 18:33:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 18:33:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:53 proc_lsst.multi:170 [INFO] [multi] job_id 17020667 proc_lsst.multi INFO: [multi] job_id 17020667 2024-03-13 18:33:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 18:33:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:53 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020667 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020667 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17020667 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-13 18:33:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-13 18:33:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 18:33:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:53 proc_lsst.multi:170 [INFO] [multi] job_id 17020668 proc_lsst.multi INFO: [multi] job_id 17020668 2024-03-13 18:33:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 18:33:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:53 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020668 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020668 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17020668 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-13 18:33:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-13 18:33:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 18:33:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:54 proc_lsst.multi:170 [INFO] [multi] job_id 17020669 proc_lsst.multi INFO: [multi] job_id 17020669 2024-03-13 18:33:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 18:33:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:54 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020669 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020669 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17020669 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-13 18:33:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-13 18:33:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 18:33:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:54 proc_lsst.multi:170 [INFO] [multi] job_id 17020670 proc_lsst.multi INFO: [multi] job_id 17020670 2024-03-13 18:33:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 18:33:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:54 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020670 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020670 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17020670 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-13 18:33:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-13 18:33:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 18:33:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95bf04ef6f0edfa9344ff7fce60a72c2 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash d8480db1fe62799f452e2fadee32ff4f parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247915800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/13b3bed6-f3ab-42f7-9301-0377f9c0090d_characterizeImage_945757_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/13b3bed6-f3ab-42f7-9301-0377f9c0090d_characterizeImage_945757_54.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/6dda49f4-adac-4661-9b27-3540837c24d8_isr_945757_54.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/6dda49f4-adac-4661-9b27-3540837c24d8_isr_945757_54.stderr 2024-03-13 18:33:55 proc_lsst.multi:170 [INFO] [multi] job_id 17020671 proc_lsst.multi INFO: [multi] job_id 17020671 2024-03-13 18:33:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 18:33:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:55 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020671 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020671 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17020671 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-13 18:33:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-13 18:33:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 18:33:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:55 proc_lsst.multi:170 [INFO] [multi] job_id 17020672 proc_lsst.multi INFO: [multi] job_id 17020672 2024-03-13 18:33:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 18:33:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:55 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020672 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020672 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17020672 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-13 18:33:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-13 18:33:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 18:33:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cccbf907f3c5a55d1c67b89b3fc51fd with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 92b131f870e5ed1933413d51fbbbe8c3 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247916200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/e832b6f3-739d-43e8-98dd-aacb42b6d297_characterizeImage_945869_5.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/e832b6f3-739d-43e8-98dd-aacb42b6d297_characterizeImage_945869_5.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e1850ac7-9092-445b-90f1-f95a6842ccf2_isr_945869_5.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e1850ac7-9092-445b-90f1-f95a6842ccf2_isr_945869_5.stderr 2024-03-13 18:33:56 proc_lsst.multi:170 [INFO] [multi] job_id 17020673 proc_lsst.multi INFO: [multi] job_id 17020673 2024-03-13 18:33:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 18:33:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:56 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020673 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020673 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17020673 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-13 18:33:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-13 18:33:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 18:33:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:56 proc_lsst.multi:170 [INFO] [multi] job_id 17020674 proc_lsst.multi INFO: [multi] job_id 17020674 2024-03-13 18:33:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 18:33:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:56 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020674 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020674 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17020674 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-13 18:33:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-13 18:33:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 18:33:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:57 proc_lsst.multi:170 [INFO] [multi] job_id 17020675 proc_lsst.multi INFO: [multi] job_id 17020675 2024-03-13 18:33:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 18:33:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:57 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020675 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020675 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17020675 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-13 18:33:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-13 18:33:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 18:33:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:57 proc_lsst.multi:170 [INFO] [multi] job_id 17020676 proc_lsst.multi INFO: [multi] job_id 17020676 2024-03-13 18:33:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 18:33:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:57 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020676 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020676 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17020676 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-13 18:33:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-13 18:33:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 18:33:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:58 proc_lsst.multi:170 [INFO] [multi] job_id 17020678 proc_lsst.multi INFO: [multi] job_id 17020678 2024-03-13 18:33:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 18:33:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:58 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020678 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020678 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17020678 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-13 18:33:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-13 18:33:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 18:33:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e94ee661bd7df06c2e36d9c819945e9 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 36c7d87840f04491b838bbc4c38c9811 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247916520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bccd4b7b-0a93-4de6-b5aa-f4a96c21ab83_characterizeImage_945869_29.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bccd4b7b-0a93-4de6-b5aa-f4a96c21ab83_characterizeImage_945869_29.stderr parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/6628ee47-f59d-458b-a784-5a1d50149b94_isr_945869_29.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/6628ee47-f59d-458b-a784-5a1d50149b94_isr_945869_29.stderr 2024-03-13 18:33:58 proc_lsst.multi:170 [INFO] [multi] job_id 17020679 proc_lsst.multi INFO: [multi] job_id 17020679 2024-03-13 18:33:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 18:33:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:58 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020679 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020679 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17020679 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-13 18:33:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-13 18:33:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 18:33:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:58 proc_lsst.multi:170 [INFO] [multi] job_id 17020680 proc_lsst.multi INFO: [multi] job_id 17020680 2024-03-13 18:33:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 18:33:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:58 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020680 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020680 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17020680 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-13 18:33:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-13 18:33:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 18:33:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:59 proc_lsst.multi:170 [INFO] [multi] job_id 17020681 proc_lsst.multi INFO: [multi] job_id 17020681 2024-03-13 18:33:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:33:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:59 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17020681 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17020681 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17020681 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-13 18:33:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:33:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:59 proc_lsst.multi:170 [INFO] [multi] job_id 17020682 proc_lsst.multi INFO: [multi] job_id 17020682 2024-03-13 18:33:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:33:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020682 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020682 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17020682 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-13 18:33:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:33:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:33:59 proc_lsst.multi:170 [INFO] [multi] job_id 17020683 proc_lsst.multi INFO: [multi] job_id 17020683 2024-03-13 18:33:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 18:33:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:33:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020683 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020683 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17020683 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-13 18:33:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:33:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:33:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 18:33:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:33:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:00 proc_lsst.multi:170 [INFO] [multi] job_id 17020684 proc_lsst.multi INFO: [multi] job_id 17020684 2024-03-13 18:34:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 18:34:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020684 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020684 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17020684 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-13 18:34:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:00 proc_lsst.multi:170 [INFO] [multi] job_id 17020685 proc_lsst.multi INFO: [multi] job_id 17020685 2024-03-13 18:34:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 18:34:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020685 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020685 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17020685 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-13 18:34:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:00 proc_lsst.multi:170 [INFO] [multi] job_id 17020686 proc_lsst.multi INFO: [multi] job_id 17020686 2024-03-13 18:34:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020686 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020686 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17020686 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-13 18:34:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:00 proc_lsst.multi:170 [INFO] [multi] job_id 17020687 proc_lsst.multi INFO: [multi] job_id 17020687 2024-03-13 18:34:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 18:34:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020687 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020687 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17020687 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-13 18:34:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 18:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:01 proc_lsst.multi:170 [INFO] [multi] job_id 17020688 proc_lsst.multi INFO: [multi] job_id 17020688 2024-03-13 18:34:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 18:34:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020688 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020688 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17020688 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-13 18:34:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:01 proc_lsst.multi:170 [INFO] [multi] job_id 17020689 proc_lsst.multi INFO: [multi] job_id 17020689 2024-03-13 18:34:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 18:34:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020689 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020689 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17020689 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-13 18:34:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:01 proc_lsst.multi:170 [INFO] [multi] job_id 17020690 proc_lsst.multi INFO: [multi] job_id 17020690 2024-03-13 18:34:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 18:34:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020690 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020690 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17020690 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-13 18:34:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:01 proc_lsst.multi:170 [INFO] [multi] job_id 17020691 proc_lsst.multi INFO: [multi] job_id 17020691 2024-03-13 18:34:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 18:34:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020691 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020691 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17020691 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-13 18:34:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 18:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:02 proc_lsst.multi:170 [INFO] [multi] job_id 17020692 proc_lsst.multi INFO: [multi] job_id 17020692 2024-03-13 18:34:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 18:34:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020692 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020692 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17020692 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-13 18:34:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:02 proc_lsst.multi:170 [INFO] [multi] job_id 17020693 proc_lsst.multi INFO: [multi] job_id 17020693 2024-03-13 18:34:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 18:34:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020693 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020693 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17020693 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-13 18:34:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:02 proc_lsst.multi:170 [INFO] [multi] job_id 17020694 proc_lsst.multi INFO: [multi] job_id 17020694 2024-03-13 18:34:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 18:34:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020694 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020694 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17020694 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-13 18:34:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:02 proc_lsst.multi:170 [INFO] [multi] job_id 17020695 proc_lsst.multi INFO: [multi] job_id 17020695 2024-03-13 18:34:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 18:34:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020695 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020695 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17020695 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-13 18:34:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:02 proc_lsst.multi:170 [INFO] [multi] job_id 17020696 proc_lsst.multi INFO: [multi] job_id 17020696 2024-03-13 18:34:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 18:34:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020696 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020696 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17020696 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-13 18:34:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 18:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:03 proc_lsst.multi:170 [INFO] [multi] job_id 17020697 proc_lsst.multi INFO: [multi] job_id 17020697 2024-03-13 18:34:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 18:34:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020697 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020697 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17020697 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-13 18:34:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:03 proc_lsst.multi:170 [INFO] [multi] job_id 17020698 proc_lsst.multi INFO: [multi] job_id 17020698 2024-03-13 18:34:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 18:34:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020698 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020698 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17020698 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-13 18:34:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:03 proc_lsst.multi:170 [INFO] [multi] job_id 17020699 proc_lsst.multi INFO: [multi] job_id 17020699 2024-03-13 18:34:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 18:34:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020699 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020699 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17020699 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-13 18:34:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:03 proc_lsst.multi:170 [INFO] [multi] job_id 17020700 proc_lsst.multi INFO: [multi] job_id 17020700 2024-03-13 18:34:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 18:34:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020700 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020700 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17020700 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-13 18:34:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 18:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:04 proc_lsst.multi:170 [INFO] [multi] job_id 17020701 proc_lsst.multi INFO: [multi] job_id 17020701 2024-03-13 18:34:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020701 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020701 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17020701 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-13 18:34:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:04 proc_lsst.multi:170 [INFO] [multi] job_id 17020702 proc_lsst.multi INFO: [multi] job_id 17020702 2024-03-13 18:34:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 18:34:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020702 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020702 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17020702 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-13 18:34:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:04 proc_lsst.multi:170 [INFO] [multi] job_id 17020703 proc_lsst.multi INFO: [multi] job_id 17020703 2024-03-13 18:34:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 18:34:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020703 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020703 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17020703 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-13 18:34:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:04 proc_lsst.multi:170 [INFO] [multi] job_id 17020704 proc_lsst.multi INFO: [multi] job_id 17020704 2024-03-13 18:34:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 18:34:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020704 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020704 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17020704 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-13 18:34:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:04 proc_lsst.multi:170 [INFO] [multi] job_id 17020705 proc_lsst.multi INFO: [multi] job_id 17020705 2024-03-13 18:34:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 18:34:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020705 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020705 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17020705 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-13 18:34:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 18:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:05 proc_lsst.multi:170 [INFO] [multi] job_id 17020706 proc_lsst.multi INFO: [multi] job_id 17020706 2024-03-13 18:34:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 18:34:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020706 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020706 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17020706 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-13 18:34:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:05 proc_lsst.multi:170 [INFO] [multi] job_id 17020707 proc_lsst.multi INFO: [multi] job_id 17020707 2024-03-13 18:34:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 18:34:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020707 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020707 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17020707 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-13 18:34:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91ecf30da07a9e05d5bed5a1330575a8 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash ea6bc891a2576428696bf27f2d063e0d parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/4633bbac-56c1-4997-897d-d025a7f50dd4_characterizeImage_945869_8.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/4633bbac-56c1-4997-897d-d025a7f50dd4_characterizeImage_945869_8.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/7ce1a1b3-1f93-4865-8022-b6981ba6bd4c_isr_945869_8.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/7ce1a1b3-1f93-4865-8022-b6981ba6bd4c_isr_945869_8.stderr 2024-03-13 18:34:05 proc_lsst.multi:170 [INFO] [multi] job_id 17020708 proc_lsst.multi INFO: [multi] job_id 17020708 2024-03-13 18:34:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 18:34:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020708 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020708 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17020708 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-13 18:34:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:05 proc_lsst.multi:170 [INFO] [multi] job_id 17020709 proc_lsst.multi INFO: [multi] job_id 17020709 2024-03-13 18:34:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 18:34:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020709 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020709 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17020709 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-13 18:34:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 18:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:06 proc_lsst.multi:170 [INFO] [multi] job_id 17020710 proc_lsst.multi INFO: [multi] job_id 17020710 2024-03-13 18:34:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 18:34:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020710 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020710 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17020710 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-13 18:34:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:06 proc_lsst.multi:170 [INFO] [multi] job_id 17020711 proc_lsst.multi INFO: [multi] job_id 17020711 2024-03-13 18:34:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020711 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020711 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17020711 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-13 18:34:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:06 proc_lsst.multi:170 [INFO] [multi] job_id 17020712 proc_lsst.multi INFO: [multi] job_id 17020712 2024-03-13 18:34:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-13 18:34:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020712 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020712 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17020712 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-13 18:34:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:06 proc_lsst.multi:170 [INFO] [multi] job_id 17020713 proc_lsst.multi INFO: [multi] job_id 17020713 2024-03-13 18:34:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-13 18:34:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020713 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020713 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17020713 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-13 18:34:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-13 18:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:07 proc_lsst.multi:170 [INFO] [multi] job_id 17020714 proc_lsst.multi INFO: [multi] job_id 17020714 2024-03-13 18:34:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-13 18:34:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020714 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020714 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17020714 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-13 18:34:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:07 proc_lsst.multi:170 [INFO] [multi] job_id 17020715 proc_lsst.multi INFO: [multi] job_id 17020715 2024-03-13 18:34:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-13 18:34:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020715 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020715 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17020715 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-13 18:34:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:07 proc_lsst.multi:170 [INFO] [multi] job_id 17020716 proc_lsst.multi INFO: [multi] job_id 17020716 2024-03-13 18:34:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-13 18:34:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020716 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020716 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17020716 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-13 18:34:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-13 18:34:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 18:34:08 proc_lsst.multi:170 [INFO] [multi] job_id 17020717 proc_lsst.multi INFO: [multi] job_id 17020717 2024-03-13 18:34:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-13 18:34:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020717 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020717 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17020717 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-13 18:34:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-13 18:34:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 18:34:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 18:34:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 18:34:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 18:34:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 18:34:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 18:34:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 18:34:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 18:34:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 18:34:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 18:34:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-13 18:34:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54444 --result_port=54385 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57d1a09c1ec7327088896bd167bb6b9e with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 9542c163eebea7e70ba1a26ce3955bd8 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/d1cc340e-fac3-4cae-ac06-5cb3da00b59d_characterizeImage_945757_40.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/d1cc340e-fac3-4cae-ac06-5cb3da00b59d_characterizeImage_945757_40.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/6a82f918-cdf5-4dce-968f-988166418a3b_isr_945757_40.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/6a82f918-cdf5-4dce-968f-988166418a3b_isr_945757_40.stderr 2024-03-13 18:34:08 proc_lsst.multi:170 [INFO] [multi] job_id 17020718 proc_lsst.multi INFO: [multi] job_id 17020718 2024-03-13 18:34:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-13 18:34:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-13 18:34:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17020718 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17020718 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17020718 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:34:08 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:34:08 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:08 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:09 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:09 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:09 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:09 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:10 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:10 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:10 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:10 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:11 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:11 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:11 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:11 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64c5605ccaa598c2bb3277cc3f853566 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 013be670eebf8f81dc114239ee55e185 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247916ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2092bf89-0583-4ffa-84b0-5b1418e1983b_characterizeImage_945869_62.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2092bf89-0583-4ffa-84b0-5b1418e1983b_characterizeImage_945869_62.stderr parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/ea174d62-ed86-46ca-8299-a78c0aa30985_isr_945869_62.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/ea174d62-ed86-46ca-8299-a78c0aa30985_isr_945869_62.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2d4b67c16171d4d6461c5c281b710f3 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 3e5626144ea023996f4e475d68e67b44 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1e1beb6c-29ed-4b9e-aa14-4ab8e05e71e4_characterizeImage_945869_4.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1e1beb6c-29ed-4b9e-aa14-4ab8e05e71e4_characterizeImage_945869_4.stderr parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/2feafb51-ca72-4661-a6b5-eaae709d1522_isr_945869_4.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/2feafb51-ca72-4661-a6b5-eaae709d1522_isr_945869_4.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abd370f213879030511aa5946e4297dc with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash c7dfd4b0345f2819d07e08d60f314e88 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247917060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/becaa19d-15b4-446f-a2c4-88ebf144089b_characterizeImage_945755_24.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/becaa19d-15b4-446f-a2c4-88ebf144089b_characterizeImage_945755_24.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/855a30c6-301b-48ad-8111-e92db41b3a87_isr_945755_24.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/855a30c6-301b-48ad-8111-e92db41b3a87_isr_945755_24.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f29130236ee59c9906ccfccfe49d2c0 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 1dd4e932acc5267908919513166f0d07 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/49893688-6cc2-43a9-953a-40a926fb68b7_characterizeImage_945869_20.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/49893688-6cc2-43a9-953a-40a926fb68b7_characterizeImage_945869_20.stderr parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/35a9409f-0acf-4fa9-8116-c5cf586602ed_isr_945869_20.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/35a9409f-0acf-4fa9-8116-c5cf586602ed_isr_945869_20.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e49d86749baa4122bcb594813e38b84 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash ec86d12a9e6cc3bad335fbce025a81b2 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/b0d7e9d4-e557-420a-a042-3b70e4c4ad08_characterizeImage_945868_23.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/b0d7e9d4-e557-420a-a042-3b70e4c4ad08_characterizeImage_945868_23.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8a25dc69-dae0-4847-bdaf-cd12a4894de0_isr_945868_23.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8a25dc69-dae0-4847-bdaf-cd12a4894de0_isr_945868_23.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d16a44d35f7f703b711b7bf334e0c03 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash d300d4743bd85db25e2bb5741fff6053 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247917920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/22c98ded-882f-438a-a980-66982690d9d7_characterizeImage_945755_23.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/22c98ded-882f-438a-a980-66982690d9d7_characterizeImage_945755_23.stderr parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/54b0bec5-ceef-4fd4-b384-3b11305de974_isr_945755_23.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/54b0bec5-ceef-4fd4-b384-3b11305de974_isr_945755_23.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37c0c5d53b6e00fbb43922ee41b19164 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 73c5b540685f22d95ef944d7c63506d7 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247917f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/6bbcf4dd-bab9-4665-bc88-e33fddd327ab_characterizeImage_945868_35.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/6bbcf4dd-bab9-4665-bc88-e33fddd327ab_characterizeImage_945868_35.stderr parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/f2cf5d78-f158-48de-9297-fc5ba1a558fb_isr_945868_35.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/f2cf5d78-f158-48de-9297-fc5ba1a558fb_isr_945868_35.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5ebd4b35b97993aca7f3bda0b657d32 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 1f9fccf85185807c513cf23f1b7cc1b5 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479168e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/56d86744-beed-4768-9cfa-eafe88c2595a_characterizeImage_945868_4.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/56d86744-beed-4768-9cfa-eafe88c2595a_characterizeImage_945868_4.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/a51d3912-9d6f-4279-bfad-0172fa492418_isr_945868_4.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/a51d3912-9d6f-4279-bfad-0172fa492418_isr_945868_4.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d43bdbba5df6c059905e7ff7e8f7721 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash dc3547ced378fd98447a14f1ca99da2a parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247917600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/cd8dc4f4-e3b7-4421-8b4b-79a808b9e9f0_characterizeImage_945867_7.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/cd8dc4f4-e3b7-4421-8b4b-79a808b9e9f0_characterizeImage_945867_7.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2764af42-aeeb-4859-b050-5e57f7a0055a_isr_945867_7.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2764af42-aeeb-4859-b050-5e57f7a0055a_isr_945867_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:12 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e619aa3ed93ce47d39a23b66442e01d with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 52b6e8afb07771b107421b0d349b490a parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247917240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/3ba911be-f824-4723-b370-91a0b41e61a9_characterizeImage_945869_53.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/3ba911be-f824-4723-b370-91a0b41e61a9_characterizeImage_945869_53.stderr parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/0b541f54-f937-4e50-850b-be875a489dad_isr_945869_53.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/0b541f54-f937-4e50-850b-be875a489dad_isr_945869_53.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfb2c902f0ef8928942c9150fb09358a with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 41b08934c51f77bb295dc4d2107b002e parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/c77b24e4-6910-4773-831a-208a9ef6e713_characterizeImage_945868_49.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/c77b24e4-6910-4773-831a-208a9ef6e713_characterizeImage_945868_49.stderr parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/716615b0-56be-4640-b9b1-f7143953cea9_isr_945868_49.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/716615b0-56be-4640-b9b1-f7143953cea9_isr_945868_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:12 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eac62ec2a975e2b5b5e37a29056348fc with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 799b455dba89a17f21e8cea74092c280 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/3db0162c-a36b-4b1e-9680-3a59e52f3c9d_characterizeImage_945755_37.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/3db0162c-a36b-4b1e-9680-3a59e52f3c9d_characterizeImage_945755_37.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/334a821e-6b0a-4170-86e8-234f929e04a5_isr_945755_37.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/334a821e-6b0a-4170-86e8-234f929e04a5_isr_945755_37.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fa4dd38b1613d740d26384afbb7f4ac with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 187dc581a90ea3a38ef5d6e367618f13 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247917c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a19cacf9-e8fc-4b26-b8d0-7d27e1d34a94_characterizeImage_945868_29.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a19cacf9-e8fc-4b26-b8d0-7d27e1d34a94_characterizeImage_945868_29.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/568fb02f-20cc-4514-a446-edccc967c773_isr_945868_29.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/568fb02f-20cc-4514-a446-edccc967c773_isr_945868_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:13 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:13 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:14 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:14 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:14 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:15 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:15 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78d1d9e6d5fb1e31c224dc731d416dcb with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash ef67b935115af5db7036e4c6d65d1f2c parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/dc240790-8a43-40e0-931b-22414e4c3038_characterizeImage_945756_55.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/dc240790-8a43-40e0-931b-22414e4c3038_characterizeImage_945756_55.stderr parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/70fabf7d-91a9-4ac0-9cf8-dc83acca4296_isr_945756_55.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/70fabf7d-91a9-4ac0-9cf8-dc83acca4296_isr_945756_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:15 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:15 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:16 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:16 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:16 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:16 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:17 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:17 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de232b03c7e1027d4596195bd1aa934f with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash c8bd9c3a01f10e208a79bf18c6fdf29b parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/cc5bd271-be5a-4243-bd4c-7d3cddfb58c2_characterizeImage_945756_11.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/cc5bd271-be5a-4243-bd4c-7d3cddfb58c2_characterizeImage_945756_11.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/b7eead58-2e80-4c9d-9fd0-5d0f95f9d347_isr_945756_11.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/b7eead58-2e80-4c9d-9fd0-5d0f95f9d347_isr_945756_11.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a66b5fc7793e565724a70fece02d57a2 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 9e598fe469cc91e599d5905e053d6b5e parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/89efe46e-bee3-4d1a-9c36-63b5c5346615_characterizeImage_945869_41.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/89efe46e-bee3-4d1a-9c36-63b5c5346615_characterizeImage_945869_41.stderr parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e2588901-0392-4c0c-9af8-eedfeb1afff7_isr_945869_41.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e2588901-0392-4c0c-9af8-eedfeb1afff7_isr_945869_41.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fb8491ea13e330d0661bb7da4a2f346 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 7f84c078aab1fef2fc64fe42e1e25891 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b84f6c29-0de1-49ec-a0dd-967112729d96_characterizeImage_945757_51.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b84f6c29-0de1-49ec-a0dd-967112729d96_characterizeImage_945757_51.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/520a0f09-31a5-4740-a159-d7ee25d25e91_isr_945757_51.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/520a0f09-31a5-4740-a159-d7ee25d25e91_isr_945757_51.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0ed62dea488147ddeeaf114ae102afa with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash d45bfde5a6f6285c3f71cd7585fdb33f parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/8f768533-dd79-4cc4-b3f0-d59f03de01ef_characterizeImage_945868_15.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/8f768533-dd79-4cc4-b3f0-d59f03de01ef_characterizeImage_945868_15.stderr parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c2a995fe-d799-4b3f-8063-2c7f677b067b_isr_945868_15.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c2a995fe-d799-4b3f-8063-2c7f677b067b_isr_945868_15.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8df3a9398552b8d607f7790a8e19d640 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 23fac2d5b15739bc31698ceb4772c530 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/9d062a4d-cdd5-40f4-ac46-61304a70fef8_characterizeImage_945868_19.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/9d062a4d-cdd5-40f4-ac46-61304a70fef8_characterizeImage_945868_19.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8130c8fc-9433-43c5-a63e-7204cab5522e_isr_945868_19.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8130c8fc-9433-43c5-a63e-7204cab5522e_isr_945868_19.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3401458397aded6038207cee4610dc60 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash c4f92cd564f8e22ba21386246b4c69a2 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/04748ef6-0ea7-4396-9651-0812ae8b7036_characterizeImage_945867_48.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/04748ef6-0ea7-4396-9651-0812ae8b7036_characterizeImage_945867_48.stderr parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/e00001b2-0d0e-435b-a8ad-a587c225bd5a_isr_945867_48.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/e00001b2-0d0e-435b-a8ad-a587c225bd5a_isr_945867_48.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 760240fb5b7b270952c95a42c9492521 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 1f2552bfc15a8e31bd313c626c512a5a parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/9c262849-fd75-4efc-88be-8b46d9183802_characterizeImage_945757_34.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/9c262849-fd75-4efc-88be-8b46d9183802_characterizeImage_945757_34.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a0099220-29ce-48a4-8f70-7b702553d321_isr_945757_34.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a0099220-29ce-48a4-8f70-7b702553d321_isr_945757_34.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 951df5cde2f0aac7b4072bcff2fbbab1 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash afa8cfd6c33f17e7ce5412da390120cf parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0e1aaf65-35df-4955-bd00-6394c384adbf_characterizeImage_945868_24.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0e1aaf65-35df-4955-bd00-6394c384adbf_characterizeImage_945868_24.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/3878e9c7-89b4-4eda-af3d-ae36970b01a9_isr_945868_24.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/3878e9c7-89b4-4eda-af3d-ae36970b01a9_isr_945868_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:17 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c61d5f9104d4d953780a6e8fb49a204a with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash b9dca989126e1e42255b3cf7028952d1 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/a3883c77-df8d-4a56-a1f5-c319121e2bae_characterizeImage_945756_26.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/a3883c77-df8d-4a56-a1f5-c319121e2bae_characterizeImage_945756_26.stderr parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/9500330a-d7a6-43c1-b380-6334db5b4a79_isr_945756_26.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/9500330a-d7a6-43c1-b380-6334db5b4a79_isr_945756_26.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5bc421a8f7e49c81198bcfe508d44f5 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash ea68f103b89fca8a0fd26cbb6c15f67d parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dedf45af-d869-4edf-9965-3981ddb78f00_characterizeImage_945757_7.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dedf45af-d869-4edf-9965-3981ddb78f00_characterizeImage_945757_7.stderr parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d7c2c130-d4e5-4228-b908-c14d76da36f1_isr_945757_7.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d7c2c130-d4e5-4228-b908-c14d76da36f1_isr_945757_7.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e009cf1b511b2a7da45fc65b8cf59090 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 3108a0baeb77a7c2424d337d0fa41961 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/2a34b4b9-cc64-4dd7-8f1c-42a353ad390f_characterizeImage_945755_55.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/2a34b4b9-cc64-4dd7-8f1c-42a353ad390f_characterizeImage_945755_55.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d58748a3-37f1-41ee-b826-001c157c0105_isr_945755_55.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d58748a3-37f1-41ee-b826-001c157c0105_isr_945755_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:18 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:18 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:18 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:19 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:19 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:19 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6abc3b2715967ad5040f6c5d78ba5eac with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash ddb0b41a1a7fec1a529d646bea248503 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/71f431f0-a1ed-42b2-9b58-8ba15910cc7a_characterizeImage_945757_55.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/71f431f0-a1ed-42b2-9b58-8ba15910cc7a_characterizeImage_945757_55.stderr parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/1b793783-5e3f-4f2e-b705-d5e26158e70a_isr_945757_55.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/1b793783-5e3f-4f2e-b705-d5e26158e70a_isr_945757_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:20 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:20 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:21 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:21 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:22 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:22 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:23 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:23 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:24 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:24 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:25 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2741ca1733a8955a695fbc932c1ee282 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 31e1cf9742012f3f089ddcaa8379f8da parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:25 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/881a4e45-8dd0-4d15-9d99-ec7bb24c3a25_characterizeImage_945869_15.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/881a4e45-8dd0-4d15-9d99-ec7bb24c3a25_characterizeImage_945869_15.stderr parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/37e02578-614f-4e56-9640-f1e69d7bdeb4_isr_945869_15.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/37e02578-614f-4e56-9640-f1e69d7bdeb4_isr_945869_15.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fc7fb31e4e28e7bb49838eab0282d65 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash a68e11777a31eef4786ab29761ad3bd6 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/1b846dea-12fb-4aa1-82fa-0795f27ef107_characterizeImage_945755_15.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/1b846dea-12fb-4aa1-82fa-0795f27ef107_characterizeImage_945755_15.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/cf0d5b35-7fb0-47bf-bcf6-d50f70851b30_isr_945755_15.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/cf0d5b35-7fb0-47bf-bcf6-d50f70851b30_isr_945755_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:26 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 074170d09172d25870ba9e0c204105f6 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 07e4daf3f8d45bafc69d03541f00b336 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/b3d46c05-8045-4775-a83e-f006ce86e7c2_characterizeImage_945869_56.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/b3d46c05-8045-4775-a83e-f006ce86e7c2_characterizeImage_945869_56.stderr parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/0f84601d-49ad-4909-a667-e5cae5c30914_isr_945869_56.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/0f84601d-49ad-4909-a667-e5cae5c30914_isr_945869_56.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f986582d6ca132719948b649bb3a2ff with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 71d961a31eed8c962d5348606a737099 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/571a2963-a91e-438c-aa6f-567dcc9eb220_characterizeImage_945757_32.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/571a2963-a91e-438c-aa6f-567dcc9eb220_characterizeImage_945757_32.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/9ff6e045-1e3c-4989-8018-214b4e9ebcde_isr_945757_32.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/9ff6e045-1e3c-4989-8018-214b4e9ebcde_isr_945757_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:26 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2a1544a90f643e075015fe6893a06bf with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 3b65fbfefe43affee4b1fd05ce20a257 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479940e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/d0248f33-9412-4e93-8dab-ce724eade571_characterizeImage_945869_10.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/d0248f33-9412-4e93-8dab-ce724eade571_characterizeImage_945869_10.stderr parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/4c9173f9-91e3-46e3-8281-6b7608cc6251_isr_945869_10.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/4c9173f9-91e3-46e3-8281-6b7608cc6251_isr_945869_10.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80fe0d8a3d2b9571149b583ea7ff8804 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 59ec6bc0193abc729608d9818242d9a3 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dadc18d9-e0c1-4d79-a580-b0b780d062b8_characterizeImage_945757_59.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dadc18d9-e0c1-4d79-a580-b0b780d062b8_characterizeImage_945757_59.stderr parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f493f01b-dd9e-4e89-9122-d722f79dc4af_isr_945757_59.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f493f01b-dd9e-4e89-9122-d722f79dc4af_isr_945757_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:27 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:27 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40cfdaa316a60e00a4c27a7aa77ebfb5 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 250fb955c66fe22e2dc22c7fc586c2dd parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0d81253e-8a90-4a1a-a098-f3c6867dc7ca_characterizeImage_945868_27.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0d81253e-8a90-4a1a-a098-f3c6867dc7ca_characterizeImage_945868_27.stderr parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8b7a13a0-3ecd-44f8-a182-ae3d97e7116a_isr_945868_27.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8b7a13a0-3ecd-44f8-a182-ae3d97e7116a_isr_945868_27.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ddd6aaea8a518b3f7d973a84adc5321 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 2f2b01cdccd5cc07e89d7b7cfba16af0 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247994cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/4a42f0e2-15bf-4839-946c-385b601327a2_characterizeImage_945757_41.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/4a42f0e2-15bf-4839-946c-385b601327a2_characterizeImage_945757_41.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b0ac4ad1-cb33-4808-a5ef-0330b498f927_isr_945757_41.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b0ac4ad1-cb33-4808-a5ef-0330b498f927_isr_945757_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:27 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7833f1a46d85408d01255367a383dd96 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 3fd8c1e952d7cb931a289f4484e211c9 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/b0abcd38-9053-4b22-8f17-1c590a9da560_characterizeImage_945868_37.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/b0abcd38-9053-4b22-8f17-1c590a9da560_characterizeImage_945868_37.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/3c1f00f3-63ba-4d43-a289-51c7fef60722_isr_945868_37.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/3c1f00f3-63ba-4d43-a289-51c7fef60722_isr_945868_37.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 341d57cf688c1c08e190a9f4ccac2fff with result from task 225 parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:27 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash fd183995066a1e19d78473f3d53fed52 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479949a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/6a9bfa78-3bad-40e3-945e-015c8196fe96_characterizeImage_945867_40.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/6a9bfa78-3bad-40e3-945e-015c8196fe96_characterizeImage_945867_40.stderr parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2f9b4c8d-833a-4127-9ae5-7a98653d084d_isr_945867_40.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2f9b4c8d-833a-4127-9ae5-7a98653d084d_isr_945867_40.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8af4c97e30ba0c2057f9dd437195d6ca with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash ce0a94af5f758105211abcbf5f98e662 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/9be45a17-d518-4b33-9712-6c19764a7247_characterizeImage_945757_56.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/9be45a17-d518-4b33-9712-6c19764a7247_characterizeImage_945757_56.stderr parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/20d67d8b-cb1a-49ce-a16f-736841bcf2db_isr_945757_56.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/20d67d8b-cb1a-49ce-a16f-736841bcf2db_isr_945757_56.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37c2557dee120c3750dbdda997e9d89c with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 755679b5de4a76e02deedc2e3a365518 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/40373139-de3b-462c-8da4-779177b77ec5_characterizeImage_945757_39.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/40373139-de3b-462c-8da4-779177b77ec5_characterizeImage_945757_39.stderr parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/1acce9a5-3095-412c-991c-fe9041b4f7e1_isr_945757_39.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/1acce9a5-3095-412c-991c-fe9041b4f7e1_isr_945757_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:28 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b5dca17209335a1565baeb0668b32a9 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 5a9ad0a016edcaec86dc2397660da797 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/d7866d19-bd70-4ea3-8d1c-c093dadc1ead_characterizeImage_945868_58.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/d7866d19-bd70-4ea3-8d1c-c093dadc1ead_characterizeImage_945868_58.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9f09fa07-9bd5-4ed6-afae-81c94ce72a9f_isr_945868_58.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9f09fa07-9bd5-4ed6-afae-81c94ce72a9f_isr_945868_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:28 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:28 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:29 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b06fb01b0e15ce9077f0ceb80df6122 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash fb9673fc39ee61d7deab98dc805a5401 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ca01fbce-00bb-4888-9056-798b54ecc109_characterizeImage_945755_52.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ca01fbce-00bb-4888-9056-798b54ecc109_characterizeImage_945755_52.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/6062b832-5b72-4a16-b900-3505ab21d07e_isr_945755_52.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/6062b832-5b72-4a16-b900-3505ab21d07e_isr_945755_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:29 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:29 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:30 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:30 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:30 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f08c42737a48cd5e384e3c3286bdbe63 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 8e6a5efd6dd6bf31edae0f52e2a997d3 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247994680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6feaebef-61d1-4fa6-a86c-a02d1586bde4_characterizeImage_945756_36.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6feaebef-61d1-4fa6-a86c-a02d1586bde4_characterizeImage_945756_36.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/23210f94-6771-4a7a-a569-7d05930f35ac_isr_945756_36.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/23210f94-6771-4a7a-a569-7d05930f35ac_isr_945756_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:30 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:31 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:31 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a62829fd628964e13ecf695739c1ffdc with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash bb94994bad1a969dfc15040e01ffe2a8 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/cd0f69e2-4f61-4c4a-9117-d292bae37df4_characterizeImage_945756_60.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/cd0f69e2-4f61-4c4a-9117-d292bae37df4_characterizeImage_945756_60.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2aaf167b-6527-462b-9a67-2b5c5b70bfff_isr_945756_60.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2aaf167b-6527-462b-9a67-2b5c5b70bfff_isr_945756_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:31 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65cdb16a72e8c60efbba8311fc39d907 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 0b4f1cff0ec6ffd52c41d97bb0088dbc parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247994fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/0da76e25-93cd-4345-8147-898b4f324eaa_characterizeImage_945867_15.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/0da76e25-93cd-4345-8147-898b4f324eaa_characterizeImage_945867_15.stderr parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/b9c84918-88e4-409f-83f3-96c953605470_isr_945867_15.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/b9c84918-88e4-409f-83f3-96c953605470_isr_945867_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:31 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f2021958bfb8e7bc006ddd7231ca933 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 9dd48043e3147debc44d48280e28a68d parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247995940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/140c153b-7d22-4435-8447-797d8d5e2c6b_characterizeImage_945757_17.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/140c153b-7d22-4435-8447-797d8d5e2c6b_characterizeImage_945757_17.stderr parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/37eb98c5-af48-41a3-8859-799bb9fbdc7f_isr_945757_17.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/37eb98c5-af48-41a3-8859-799bb9fbdc7f_isr_945757_17.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c840ffa375daac2c136a635d1879ee8c with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash a6a73372d4be802b240fa769993a5daf parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247996f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/5959f88e-dda5-40f8-aa21-b2bc8fb54de1_characterizeImage_945867_17.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/5959f88e-dda5-40f8-aa21-b2bc8fb54de1_characterizeImage_945867_17.stderr parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/db837592-ceca-4c75-9171-79f3da080deb_isr_945867_17.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/db837592-ceca-4c75-9171-79f3da080deb_isr_945867_17.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1602d816f6f587a406c6fae2f6794bc6 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash eae2ed32b43fe7d078fd8f8d942295c2 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247995620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ea4537d9-328f-48bb-906b-cc877078c3da_characterizeImage_945867_46.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ea4537d9-328f-48bb-906b-cc877078c3da_characterizeImage_945867_46.stderr parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/ad6d2f8b-3f1d-4106-9ed2-8bb34985e13d_isr_945867_46.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/ad6d2f8b-3f1d-4106-9ed2-8bb34985e13d_isr_945867_46.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cc33adfbb116ff5d628adf24bf8358b with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash b95fbb279f3d9b9ac556e081f0fbc7d5 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247995f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/32abffb1-a804-4322-9f63-616a8b4f5dda_characterizeImage_945867_56.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/32abffb1-a804-4322-9f63-616a8b4f5dda_characterizeImage_945867_56.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/d8a6d391-a68c-4e45-8f31-992773b48f4d_isr_945867_56.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/d8a6d391-a68c-4e45-8f31-992773b48f4d_isr_945867_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:32 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 983b1fab660916c4b3965032fc4fc782 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash a756e259f8bf4ab15e917b2320b090e9 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247995c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e2e8556a-8ead-4014-8072-023d806b35d0_characterizeImage_945867_42.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e2e8556a-8ead-4014-8072-023d806b35d0_characterizeImage_945867_42.stderr parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/275ad293-2145-40a3-a83e-ee2a0a9e9485_isr_945867_42.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/275ad293-2145-40a3-a83e-ee2a0a9e9485_isr_945867_42.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f20152eb276b0dceb39fd868396c11ff with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 24fb349e980839d72397f56521b71a73 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479962a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b9e482da-6de9-49c2-a42c-e19fc2f096d0_characterizeImage_945757_35.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b9e482da-6de9-49c2-a42c-e19fc2f096d0_characterizeImage_945757_35.stderr parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/435c45eb-a359-4abe-8643-4c17158853bf_isr_945757_35.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/435c45eb-a359-4abe-8643-4c17158853bf_isr_945757_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:32 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e030f2747e0b39823269643fd533b799 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 11e260a2aa7a9d2d0b2c51105772e889 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247995300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/e063bd3a-90ff-428b-9d31-f8c3db13ee72_characterizeImage_945757_26.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/e063bd3a-90ff-428b-9d31-f8c3db13ee72_characterizeImage_945757_26.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b2230e79-109c-49b2-bdfa-1c9e367f93db_isr_945757_26.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b2230e79-109c-49b2-bdfa-1c9e367f93db_isr_945757_26.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86d17962a77e318c7aa377d8ffa8ae65 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 8618e084036e335760e6970f2c3e20d6 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247996c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/305c67b2-59a3-43d0-a731-3b75c848f947_characterizeImage_945868_56.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/305c67b2-59a3-43d0-a731-3b75c848f947_characterizeImage_945868_56.stderr parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ecccf29d-a108-44a0-b624-618879e7fc8d_isr_945868_56.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ecccf29d-a108-44a0-b624-618879e7fc8d_isr_945868_56.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fc4b7c61129ef4659726908e4cf8780 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash c1722371e49775958a89ae6885d00b13 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/5529da29-41df-465c-b6e8-fcb0d8064e36_characterizeImage_945867_37.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/5529da29-41df-465c-b6e8-fcb0d8064e36_characterizeImage_945867_37.stderr parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1601abcc-244c-4b4c-873e-d630e0f88df5_isr_945867_37.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1601abcc-244c-4b4c-873e-d630e0f88df5_isr_945867_37.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1b16ed221e283ed091f59be94d79d15 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash d001cdd4f4750a709635e1675ff97383 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/9caee69b-d1b6-4b2f-8e91-d8e482a6e859_characterizeImage_945867_36.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/9caee69b-d1b6-4b2f-8e91-d8e482a6e859_characterizeImage_945867_36.stderr parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/90f9e654-017d-451c-967a-6fdcffff674c_isr_945867_36.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/90f9e654-017d-451c-967a-6fdcffff674c_isr_945867_36.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff30ac4bff00a19dc132f99747c19a4b with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 9b296c3265ac177d46ab3db73e164dd7 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/699acbed-ec41-4e6a-a585-6a294ce9b24f_characterizeImage_945755_2.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/699acbed-ec41-4e6a-a585-6a294ce9b24f_characterizeImage_945755_2.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d45f4bcf-91ee-42a3-a921-e4170d81cb00_isr_945755_2.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d45f4bcf-91ee-42a3-a921-e4170d81cb00_isr_945755_2.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2bf5683cb795afa32313250ffb63f26 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash a236adcc87c8a116c586e314b6cc154d parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/1b1172e6-671c-44ce-be16-a76425aeb6b7_characterizeImage_945757_22.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/1b1172e6-671c-44ce-be16-a76425aeb6b7_characterizeImage_945757_22.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/bb3c2a6c-ce7b-4362-8a5f-c3d16a576d12_isr_945757_22.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/bb3c2a6c-ce7b-4362-8a5f-c3d16a576d12_isr_945757_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:33 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ca4cf6c9c9ea99873132f2a275f8a1a with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash e75d15b94dc422c87ba00f6e837893a3 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479968e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/d6e3260b-8f24-4b07-911f-8dc02222f6cc_characterizeImage_945755_62.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/d6e3260b-8f24-4b07-911f-8dc02222f6cc_characterizeImage_945755_62.stderr parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/25285027-166e-4f9b-978b-3b74e024c4e6_isr_945755_62.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/25285027-166e-4f9b-978b-3b74e024c4e6_isr_945755_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:33 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2767198565a002be6a6f607d37b3f80 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash c9a5f7e29a9006d7ab894098c77ba941 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/69615b87-5512-47a0-a45a-56852204776f_characterizeImage_945869_27.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/69615b87-5512-47a0-a45a-56852204776f_characterizeImage_945869_27.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/17b72214-7865-40f2-af1c-60b18cfdf995_isr_945869_27.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/17b72214-7865-40f2-af1c-60b18cfdf995_isr_945869_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:34 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c76856dad52c164cf0955caa1ed6727 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 4e088afbd992601a56b294d47c4ce630 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479965c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/80655bc2-b7f4-4a4a-aeec-0ffdbb0e292d_characterizeImage_945868_61.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/80655bc2-b7f4-4a4a-aeec-0ffdbb0e292d_characterizeImage_945868_61.stderr parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/90f30185-a306-4462-9d85-60bad92c968e_isr_945868_61.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/90f30185-a306-4462-9d85-60bad92c968e_isr_945868_61.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54776e958db437b7f7cf4681f6a81319 with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 1a6b699338837e8cd544c85e7056f5c9 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/93204443-7c8d-406a-a4db-41e12492d800_characterizeImage_945869_46.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/93204443-7c8d-406a-a4db-41e12492d800_characterizeImage_945869_46.stderr parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f3422e49-6e2d-4220-9cd9-4279b536f08d_isr_945869_46.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f3422e49-6e2d-4220-9cd9-4279b536f08d_isr_945869_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:34 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbf3b018691645fad93d051d109a1760 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 27ab555cecedab07cd914481fd6a7296 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/b75a3343-fdaf-46ea-8ad4-53cd58148dc2_characterizeImage_945869_25.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/b75a3343-fdaf-46ea-8ad4-53cd58148dc2_characterizeImage_945869_25.stderr parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8206b796-9922-4405-ba9f-d10c4134f0d9_isr_945869_25.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8206b796-9922-4405-ba9f-d10c4134f0d9_isr_945869_25.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50e31a0dd53520c13d623c74d0de8951 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash bc83c07ae072f52f344388d1719d22a7 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bdd4308d-5eab-4222-8a5d-3382a38f5b26_characterizeImage_945869_51.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bdd4308d-5eab-4222-8a5d-3382a38f5b26_characterizeImage_945869_51.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/2d0e8ccb-bf49-4012-83f7-bc62eda4f091_isr_945869_51.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/2d0e8ccb-bf49-4012-83f7-bc62eda4f091_isr_945869_51.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c1bc45ac2c20e930abfa3237f9cf9be with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash a6f1cad140ded797cc60dc73dccf5902 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/ddd3d83f-8dfc-4b6e-9eeb-b58e868de374_characterizeImage_945757_14.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/ddd3d83f-8dfc-4b6e-9eeb-b58e868de374_characterizeImage_945757_14.stderr parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/bb2ad845-20de-45d4-b9fd-dff5d1a24a4e_isr_945757_14.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/bb2ad845-20de-45d4-b9fd-dff5d1a24a4e_isr_945757_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:34 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:34 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:34 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:35 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:35 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:35 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:35 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db7cfd1e9c3ee7204d7d7e699aefe44c with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 7548b14687789ac10e31fcdf8f2ae10c parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/3cdbc5df-88d4-49dd-97d5-6a59acc2cb09_characterizeImage_945756_40.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/3cdbc5df-88d4-49dd-97d5-6a59acc2cb09_characterizeImage_945756_40.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/e8251338-0caa-4265-a6b8-1bf59dab0dbf_isr_945756_40.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/e8251338-0caa-4265-a6b8-1bf59dab0dbf_isr_945756_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:36 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:36 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fa174848abb0c5418fd5b144e37aa8c with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 3eb31087554ccbb5fbb2d03b2c7c8bc1 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/e3ceb59e-fc1a-4fab-806e-7f794cc06c3f_characterizeImage_945869_9.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/e3ceb59e-fc1a-4fab-806e-7f794cc06c3f_characterizeImage_945869_9.stderr parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/dab39a29-ef94-44be-b4a5-f207e0f6bb03_isr_945869_9.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/dab39a29-ef94-44be-b4a5-f207e0f6bb03_isr_945869_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:36 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:36 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:37 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:37 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:37 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:38 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:38 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 PD 17020717 PD 17020718 PD 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 90/3 running/pending blocks, and 297 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:34:38 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:34:38 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:39 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:39 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:39 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f0e9c031edfa6f275501cee3d25a46c with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash f3988e94414b91b149017674a9a901b0 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/a9e8f11c-ed4c-48e2-ad31-8746ccffc444_characterizeImage_945755_7.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/a9e8f11c-ed4c-48e2-ad31-8746ccffc444_characterizeImage_945755_7.stderr parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/aa55f5ae-6c78-4e59-a789-4bbcdf22463a_isr_945755_7.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/aa55f5ae-6c78-4e59-a789-4bbcdf22463a_isr_945755_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:40 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:40 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:40 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ac8ea4f946e838bf70d521b5641ce42 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 829ad22e6bc4ba543df740f9e1210187 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/51c6be92-0fe3-4021-b662-6bf04ced4012_characterizeImage_945868_43.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/51c6be92-0fe3-4021-b662-6bf04ced4012_characterizeImage_945868_43.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9fa1bfec-01d2-4291-b7de-64d123f9fc77_isr_945868_43.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9fa1bfec-01d2-4291-b7de-64d123f9fc77_isr_945868_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:41 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:41 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:42 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8e4360543e2cad05ad6e8073b3a55cd with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash f56c04452dbb40259972d155f9f6d842 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492464011c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/7daa1979-0ee7-4edd-960c-ac19b18b1d5b_characterizeImage_945869_40.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/7daa1979-0ee7-4edd-960c-ac19b18b1d5b_characterizeImage_945869_40.stderr parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8930ca1a-fd73-4c03-bee0-0d524622b2c5_isr_945869_40.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8930ca1a-fd73-4c03-bee0-0d524622b2c5_isr_945869_40.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68699e1b6471980b2c5aeedddf8dd709 with result from task 267 parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:42 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 5432275f2365fcde34a580f459f551fb parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/657eae25-6289-49b1-9dcd-96c39e272493_characterizeImage_945756_19.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/657eae25-6289-49b1-9dcd-96c39e272493_characterizeImage_945756_19.stderr parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/de16d5e0-be22-4403-82c2-b46a9da83182_isr_945756_19.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/de16d5e0-be22-4403-82c2-b46a9da83182_isr_945756_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:43 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2d844416bd0d38878791ea713d8dcb1 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 9c3ffa49e1ccea0a760cd5fac505a87f parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246391b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ff8cffaa-a1ee-4ed3-b332-12a70070d9d3_characterizeImage_945756_35.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ff8cffaa-a1ee-4ed3-b332-12a70070d9d3_characterizeImage_945756_35.stderr parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/f9986f8f-57f5-4ab9-9fb9-3aad74cc4c06_isr_945756_35.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/f9986f8f-57f5-4ab9-9fb9-3aad74cc4c06_isr_945756_35.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b77142b43c7ceed187db6108c80e26fd with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash daa8ca8755883e94bf8e797c7b040c99 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/2c414687-7bdf-4d59-9e50-ce61852c579d_characterizeImage_945757_20.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/2c414687-7bdf-4d59-9e50-ce61852c579d_characterizeImage_945757_20.stderr parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/2bea1c82-4da0-4cfa-825e-794273f4f9ff_isr_945757_20.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/2bea1c82-4da0-4cfa-825e-794273f4f9ff_isr_945757_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:43 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a43217d925e0aa653e0639bb7d45fd10 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash cd5cc0b386da2bb1bd4672333af248a0 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/0b584a4d-b5f9-4797-86c2-28ec77c4d646_characterizeImage_945757_1.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/0b584a4d-b5f9-4797-86c2-28ec77c4d646_characterizeImage_945757_1.stderr parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a39e3c74-1523-4301-9d8d-7e24adaa9a18_isr_945757_1.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a39e3c74-1523-4301-9d8d-7e24adaa9a18_isr_945757_1.stderr parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 560c25a7f4f7a6d419031164bcf9fb54 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash d32703b5b24a3a657a490d435fdca047 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4a1dc1f8-9dec-44c9-afda-fe391247040e_characterizeImage_945756_56.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4a1dc1f8-9dec-44c9-afda-fe391247040e_characterizeImage_945756_56.stderr parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0330dcdc-d82c-4d96-bf8f-69a76d7f75f3_isr_945756_56.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0330dcdc-d82c-4d96-bf8f-69a76d7f75f3_isr_945756_56.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4e27bb48c7ba7382bdb6f7045bca157 with result from task 420 parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:44 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash a144e8eeeff534603a064443a5e0fc8f parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/4c5bc8c7-0b72-472b-86d7-4130c9c5387b_characterizeImage_945755_19.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/4c5bc8c7-0b72-472b-86d7-4130c9c5387b_characterizeImage_945755_19.stderr parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/bcb05fb3-222a-45f8-97da-835483d65abc_isr_945755_19.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/bcb05fb3-222a-45f8-97da-835483d65abc_isr_945755_19.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32557df8f0cf5ec70dfa3b069000e418 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 5ad74fa4d5fdace4f00b73bc5c233924 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4cb75670-e905-49cf-9f6b-e6fe3664faed_characterizeImage_945867_57.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4cb75670-e905-49cf-9f6b-e6fe3664faed_characterizeImage_945867_57.stderr parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1ceaf3ba-aac3-4502-97d9-a47af4188a2f_isr_945867_57.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1ceaf3ba-aac3-4502-97d9-a47af4188a2f_isr_945867_57.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3989e486e6f7d698ad855cd96193b89f with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 008d81f8b4f4f83fd4cb377049612dc3 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/6ff15723-3f8e-4a40-8916-2df8e32bdc03_characterizeImage_945755_32.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/6ff15723-3f8e-4a40-8916-2df8e32bdc03_characterizeImage_945755_32.stderr parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3e89c763-eb4d-438b-8ae2-cde9d8ec057d_isr_945755_32.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3e89c763-eb4d-438b-8ae2-cde9d8ec057d_isr_945755_32.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea01b32649ee54911fe3e22c8319bb1c with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 503fae3b4233ddff03fb64b98438fc5a parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246390ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/2002195d-9d6b-4ab2-8c3a-f15887be3e2a_characterizeImage_945868_50.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/2002195d-9d6b-4ab2-8c3a-f15887be3e2a_characterizeImage_945868_50.stderr parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/befb6d0e-1ecf-4ea4-90af-be0c806d84aa_isr_945868_50.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/befb6d0e-1ecf-4ea4-90af-be0c806d84aa_isr_945868_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:44 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b26f623e554e746b31ede39ccfec9a4 with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 1a8208acd2e328d558d9e01f1ec5b1f5 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246391800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ed18a17c-6bcd-4107-a214-74c69cdd73e9_characterizeImage_945755_39.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ed18a17c-6bcd-4107-a214-74c69cdd73e9_characterizeImage_945755_39.stderr parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/ca46f251-0cae-4928-9ce6-e2f88aeb8092_isr_945755_39.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/ca46f251-0cae-4928-9ce6-e2f88aeb8092_isr_945755_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:45 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:45 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dabb3e97728a9059d671768853b697bf with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 4046f29317ec3c3ce9ca8f5eedd387ec parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b580f608-e6eb-4bc5-899a-8c696cdb6358_characterizeImage_945756_30.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b580f608-e6eb-4bc5-899a-8c696cdb6358_characterizeImage_945756_30.stderr parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/6405c211-b0a4-49bf-8139-e20e47ad1e3c_isr_945756_30.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/6405c211-b0a4-49bf-8139-e20e47ad1e3c_isr_945756_30.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a637c1a3892c824b4257cbc8137163d4 with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 69ac604a2186e3c014f2d764aa1b2d98 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/453eaf01-4ca6-4f5a-95df-6a311f5d3a18_characterizeImage_945756_46.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/453eaf01-4ca6-4f5a-95df-6a311f5d3a18_characterizeImage_945756_46.stderr parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/60825274-930c-48a1-8a2e-ebce681bc557_isr_945756_46.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/60825274-930c-48a1-8a2e-ebce681bc557_isr_945756_46.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 279c4a891f1c49cf1564599cacef2567 with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 56476a9e04e6d7f7bc2d9e3a61f84796 parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/df3c17b8-529c-4f12-b58c-d81aeafadefc_characterizeImage_945755_56.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/df3c17b8-529c-4f12-b58c-d81aeafadefc_characterizeImage_945755_56.stderr parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/a3bf4994-006e-428a-907a-69d4d57b072c_isr_945755_56.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/a3bf4994-006e-428a-907a-69d4d57b072c_isr_945755_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:46 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f176da890936f32240787d885493cc58 with result from task 882 parsl.dataflow.dflow DEBUG: Task 882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 883 has memoization hash d4adcd9e3a3f9e3c3f8a51a62ab9345f parsl.dataflow.memoization INFO: Task 883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461abc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/9c91aef9-a5ed-42a2-96d2-ef9c4a5bf32f_characterizeImage_945868_41.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/9c91aef9-a5ed-42a2-96d2-ef9c4a5bf32f_characterizeImage_945868_41.stderr parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/0a030639-153c-4352-ba9d-92509604fba5_isr_945868_41.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/0a030639-153c-4352-ba9d-92509604fba5_isr_945868_41.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83d473bacce1d4cedc4bf97a036de2b2 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 8798fe99383f3e106920ca735599a4f6 parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246311d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/564be45a-e7bd-401d-93b7-91bb071613da_characterizeImage_945869_33.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/564be45a-e7bd-401d-93b7-91bb071613da_characterizeImage_945869_33.stderr parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/b7bd6a74-ee93-4073-bdb2-8faddc43495c_isr_945869_33.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/b7bd6a74-ee93-4073-bdb2-8faddc43495c_isr_945869_33.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09f9203defd8f436654ef6dd49ab2bed with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 1b8a952a5a3f483cd741023bb8fdf4f1 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246392ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/1d7b610d-3401-43de-9362-91339162d783_characterizeImage_945868_17.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/1d7b610d-3401-43de-9362-91339162d783_characterizeImage_945868_17.stderr parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/6a8eb84e-bfb2-434b-9aac-4a079169b7ab_isr_945868_17.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/6a8eb84e-bfb2-434b-9aac-4a079169b7ab_isr_945868_17.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc6ec2f8158a164d5a994ccd67d2de3e with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash dbb286924fefc21cd3a8635dce3ce3c3 parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463114e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ce85a6a2-2c82-4e6f-85c4-aee266e25f11_characterizeImage_945755_35.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ce85a6a2-2c82-4e6f-85c4-aee266e25f11_characterizeImage_945755_35.stderr parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/74ad9c98-feb6-43ab-bca3-068a016aa798_isr_945755_35.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/74ad9c98-feb6-43ab-bca3-068a016aa798_isr_945755_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:46 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c2ac4b5627d2af83487738303d90ad2 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash f5498eca220284bae173bf62c6f00aa9 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/f8113798-c437-4bea-bdeb-ed6dc3167b70_characterizeImage_945867_9.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/f8113798-c437-4bea-bdeb-ed6dc3167b70_characterizeImage_945867_9.stderr parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/6b125acb-fe7e-4241-9ddb-202f06ee9820_isr_945867_9.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/6b125acb-fe7e-4241-9ddb-202f06ee9820_isr_945867_9.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a808d864dd2220aa9037cb562866a35 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 1f13f8f9cbdd26a74b712b603985950b parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/ddf7ee27-af2d-4363-80dd-85037b445efb_characterizeImage_945757_8.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/ddf7ee27-af2d-4363-80dd-85037b445efb_characterizeImage_945757_8.stderr parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b6810ff9-46bb-457a-9eb2-99af8d4dad17_isr_945757_8.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b6810ff9-46bb-457a-9eb2-99af8d4dad17_isr_945757_8.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b1f619092dc27fdf84876605f6af6b7 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash d8b128389584cdbf25a453c06a75f4b7 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/823f1ba0-fc36-44fc-846c-56831b3ed019_characterizeImage_945757_49.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/823f1ba0-fc36-44fc-846c-56831b3ed019_characterizeImage_945757_49.stderr parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b01fa0cd-e0cc-456a-a4ad-847a8ef3ad93_isr_945757_49.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/b01fa0cd-e0cc-456a-a4ad-847a8ef3ad93_isr_945757_49.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 411ac38f93924c17b937e0a7e2eec688 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash ab4632a424ad3db76f806f61db48d96a parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2f3b6492-8334-4710-8723-08ee991967ae_characterizeImage_945867_2.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2f3b6492-8334-4710-8723-08ee991967ae_characterizeImage_945867_2.stderr parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8b1bc25c-0f71-4535-accb-29ca454aef38_isr_945867_2.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8b1bc25c-0f71-4535-accb-29ca454aef38_isr_945867_2.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 130ea5583e3c8e7e007afad4ab0d0707 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 9c2ec4040785683e5125a248e7654959 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1519bd68-81d2-48cc-86e5-9b0d00beccc2_characterizeImage_945869_48.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1519bd68-81d2-48cc-86e5-9b0d00beccc2_characterizeImage_945869_48.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/37907bcf-16b3-412d-bc75-599938697f8d_isr_945869_48.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/37907bcf-16b3-412d-bc75-599938697f8d_isr_945869_48.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7663c1820e80d88d4573cfc67426d8e with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 39cc9ecc58055232dfae4c72fa6d8004 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4552cfe4-9246-4a5f-9e8a-84d939b182ff_characterizeImage_945867_21.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4552cfe4-9246-4a5f-9e8a-84d939b182ff_characterizeImage_945867_21.stderr parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/aea6b6a9-7968-41e3-a509-f864a74b99a9_isr_945867_21.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/aea6b6a9-7968-41e3-a509-f864a74b99a9_isr_945867_21.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4b6283de55b3791e1a84356f1f59430 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 9678285b0456bfff96cb74197336f16c parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/f765694f-cd94-4cce-bedf-3a21ea29e1eb_characterizeImage_945867_27.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/f765694f-cd94-4cce-bedf-3a21ea29e1eb_characterizeImage_945867_27.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2d731711-e69c-4338-bc73-c5bc7f608f1d_isr_945867_27.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2d731711-e69c-4338-bc73-c5bc7f608f1d_isr_945867_27.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df9b45bd95a3164487f7ccdb926be953 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 76148e1675a083d9dc287c4f3d8aaa63 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e0f148f9-ade4-42d3-8b16-418e4dd16d27_characterizeImage_945867_30.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e0f148f9-ade4-42d3-8b16-418e4dd16d27_characterizeImage_945867_30.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/345b8026-6065-420e-af1d-d0527e11b890_isr_945867_30.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/345b8026-6065-420e-af1d-d0527e11b890_isr_945867_30.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 791e723a82fb04de22f56b9a1f31dd60 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 0e905210b0f01681c852aca890dac166 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/8e0faa55-d9fe-4f32-bbb5-3ce52383b816_characterizeImage_945867_45.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/8e0faa55-d9fe-4f32-bbb5-3ce52383b816_characterizeImage_945867_45.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/12fb4da3-6bec-4479-86b7-0db1aa950526_isr_945867_45.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/12fb4da3-6bec-4479-86b7-0db1aa950526_isr_945867_45.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5114f08b3b4ddbcff4b45e315ad62ff with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash faafa7835d1449b574a99785850f838c parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/1ac5cc67-619f-4e0e-b294-b7fe79981361_characterizeImage_945757_50.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/1ac5cc67-619f-4e0e-b294-b7fe79981361_characterizeImage_945757_50.stderr parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f25cbaf3-3cbf-41b0-921f-651950182b2c_isr_945757_50.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f25cbaf3-3cbf-41b0-921f-651950182b2c_isr_945757_50.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ebf206d76a85dd46c6af0a365237864 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 1502e909e47a4284e023602247c3639c parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463914e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1de467c6-2bd2-4ad1-97f3-a11a138c10eb_characterizeImage_945869_26.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1de467c6-2bd2-4ad1-97f3-a11a138c10eb_characterizeImage_945869_26.stderr parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/63749640-5019-4166-b298-aa9d7b1fd6b6_isr_945869_26.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/63749640-5019-4166-b298-aa9d7b1fd6b6_isr_945869_26.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42f45cebc9d26ad38c35d188061e3b9b with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 2a067d04b4d3622abe2365664c2e94ff parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/077ab6b2-4070-4081-a6d0-55107de2176e_characterizeImage_945757_62.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/077ab6b2-4070-4081-a6d0-55107de2176e_characterizeImage_945757_62.stderr parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/52d6697c-a763-4a0e-bf21-6e3edcb0606e_isr_945757_62.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/52d6697c-a763-4a0e-bf21-6e3edcb0606e_isr_945757_62.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25b5151793102a7e67adfc177caeb37f with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 05016c5a6a4b6754b865f1c9e232839c parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/6ee307cd-4429-46c4-9b27-4331d6cecaea_characterizeImage_945867_11.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/6ee307cd-4429-46c4-9b27-4331d6cecaea_characterizeImage_945867_11.stderr parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/91a4e7a9-2c33-4bfc-821c-ec43e7c63a6a_isr_945867_11.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/91a4e7a9-2c33-4bfc-821c-ec43e7c63a6a_isr_945867_11.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cd168ba5457f63de5dae6745a290668 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash ee922c93e177a06e53d2c83c431871e2 parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246391e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/8af5bcc5-691b-41ea-bdd3-ac80c7c49832_characterizeImage_945756_38.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/8af5bcc5-691b-41ea-bdd3-ac80c7c49832_characterizeImage_945756_38.stderr parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/932e36b6-de5b-44c1-8fd4-470b2da7e3e9_isr_945756_38.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/932e36b6-de5b-44c1-8fd4-470b2da7e3e9_isr_945756_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:46 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42db85f1984e3958e54029c3ccfc379b with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash bc51b6a096aa02a0b07e69149019a44d parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246390540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/09eda586-77a6-44ef-8c34-b370f7a2f02d_characterizeImage_945757_5.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/09eda586-77a6-44ef-8c34-b370f7a2f02d_characterizeImage_945757_5.stderr parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d4bd6a9d-01b4-4f37-a3e5-6135aa0edff9_isr_945757_5.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d4bd6a9d-01b4-4f37-a3e5-6135aa0edff9_isr_945757_5.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f020fff9c68793b2ca6755f79fbccac with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash bc5b24cadef89889098480cd0a605717 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246392480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/ef599eda-3498-4034-bc62-c052e9bdc925_characterizeImage_945868_12.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/ef599eda-3498-4034-bc62-c052e9bdc925_characterizeImage_945868_12.stderr parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/dcc2269f-b3f7-4b53-9dc5-7998b419274a_isr_945868_12.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/dcc2269f-b3f7-4b53-9dc5-7998b419274a_isr_945868_12.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da0412728287969519f8840885b19497 with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash d0bdc9d66a915e4feb119a12b5ba68b4 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2b746d7f-f3ad-4d82-a086-b1cbaeda084b_characterizeImage_945869_30.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2b746d7f-f3ad-4d82-a086-b1cbaeda084b_characterizeImage_945869_30.stderr parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/b8c27e31-c304-4891-976c-cd625abb55bc_isr_945869_30.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/b8c27e31-c304-4891-976c-cd625abb55bc_isr_945869_30.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da11c63b9f3260167ff0d7c2a61697f9 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 749afe1e692691ca647fdf193d0a4a50 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246392160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/340b3d66-bcaa-4656-9fc8-997468df3cf5_characterizeImage_945868_53.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/340b3d66-bcaa-4656-9fc8-997468df3cf5_characterizeImage_945868_53.stderr parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/4627d01a-47ea-4b70-a44d-74618a2ab684_isr_945868_53.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/4627d01a-47ea-4b70-a44d-74618a2ab684_isr_945868_53.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 020e9f00768988927f5cff16c8047640 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash e57adb3b4ea0ebe7209059579da10107 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/1c5c6d88-0696-49a2-bfca-60feebe6aca4_characterizeImage_945867_5.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/1c5c6d88-0696-49a2-bfca-60feebe6aca4_characterizeImage_945867_5.stderr parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4a04a615-9c12-488b-bfe9-2fbe44b1b570_isr_945867_5.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4a04a615-9c12-488b-bfe9-2fbe44b1b570_isr_945867_5.stderr parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95d789efe74f6b11dd58951d19ad58bd with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash f930603d1d898e97b4d42d69aba8a059 parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246311760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b8121dbb-b777-485f-8167-c011549760db_characterizeImage_945757_6.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b8121dbb-b777-485f-8167-c011549760db_characterizeImage_945757_6.stderr parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/46ca6f67-28d6-4a25-bc90-768dea4203dc_isr_945757_6.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/46ca6f67-28d6-4a25-bc90-768dea4203dc_isr_945757_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:47 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 149ed09d38fcb41429a65d7a61776955 with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 59b69a0c97a854a78497d0c604732c24 parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/99471206-f7fd-4b38-99d6-03f8c5cd105f_characterizeImage_945868_52.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/99471206-f7fd-4b38-99d6-03f8c5cd105f_characterizeImage_945868_52.stderr parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c65167d5-83c3-4560-9396-98b28ea9efa2_isr_945868_52.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c65167d5-83c3-4560-9396-98b28ea9efa2_isr_945868_52.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5909439c3adc5332c6432008516bf3b with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash c47dce5c6e340ebf50605f31f1aa97e0 parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/50ca3dfd-051d-4e1b-bf18-aa0c87c6e448_characterizeImage_945867_20.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/50ca3dfd-051d-4e1b-bf18-aa0c87c6e448_characterizeImage_945867_20.stderr parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1fdd0d15-b4aa-4bdd-a30b-04cbba44063a_isr_945867_20.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1fdd0d15-b4aa-4bdd-a30b-04cbba44063a_isr_945867_20.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff826a1b7d5075bf9b78ca92942d7478 with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 54a81d3a4a76d84f73736653d9b2aee9 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/647d38e2-052f-4260-973a-7e2cef2db04b_characterizeImage_945869_60.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/647d38e2-052f-4260-973a-7e2cef2db04b_characterizeImage_945869_60.stderr parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/1d7b01e8-3e9c-4bfb-8578-c5f7cc8d6261_isr_945869_60.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/1d7b01e8-3e9c-4bfb-8578-c5f7cc8d6261_isr_945869_60.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27b2d2c5c350844076ab072aacf666d9 with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 33113f3e3a59a438d2e8145e1258c086 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463927a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/3a9c1e61-0c0f-4e33-8cae-b1c4b86c3472_characterizeImage_945755_1.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/3a9c1e61-0c0f-4e33-8cae-b1c4b86c3472_characterizeImage_945755_1.stderr parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/22f4b161-dbc2-4945-8268-b1fb7c6207eb_isr_945755_1.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/22f4b161-dbc2-4945-8268-b1fb7c6207eb_isr_945755_1.stderr parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6342ca7202dc79bad0210c33f0987522 with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash 6532e09303d6190c3fced0dc4f6c85e1 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246311da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ac738be0-ef18-4266-a784-94c24406f72c_characterizeImage_945867_16.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ac738be0-ef18-4266-a784-94c24406f72c_characterizeImage_945867_16.stderr parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/57d3e197-d16b-47b9-b014-aeaaf17cf744_isr_945867_16.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/57d3e197-d16b-47b9-b014-aeaaf17cf744_isr_945867_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:47 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0996963feb0d7af671e2900dd4ae595c with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash 9a527d3c0d88a6646f7a07647ecf27ee parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/874e4591-476b-4ff5-8c6e-e8519ea1a817_characterizeImage_945755_6.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/874e4591-476b-4ff5-8c6e-e8519ea1a817_characterizeImage_945755_6.stderr parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/cb6474f6-96bd-4d7b-b469-e5ccc67af65e_isr_945755_6.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/cb6474f6-96bd-4d7b-b469-e5ccc67af65e_isr_945755_6.stderr parsl.dataflow.dflow INFO: Task 963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf35cb6bfa4f6683910a374d18f43855 with result from task 963 parsl.dataflow.dflow DEBUG: Task 963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 964 has memoization hash 79ec8be402a45bd26f34a4ff2c53f51e parsl.dataflow.memoization INFO: Task 964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/966446c6-7a4d-41b2-b353-7e0c9421f0c5_characterizeImage_945757_24.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/966446c6-7a4d-41b2-b353-7e0c9421f0c5_characterizeImage_945757_24.stderr parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/5ce13f3a-17f8-4d70-85d7-091a4d85ab51_isr_945757_24.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/5ce13f3a-17f8-4d70-85d7-091a4d85ab51_isr_945757_24.stderr parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85f5e8030cfe3a21427a9ff48ca7d6b7 with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 6579d1565a2969544d97367c86325b35 parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246311a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1375f4dc-3e17-4d19-b71b-e0ae572d84af_characterizeImage_945869_50.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1375f4dc-3e17-4d19-b71b-e0ae572d84af_characterizeImage_945869_50.stderr parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/231f7a77-be12-4292-a8ee-527af7270885_isr_945869_50.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/231f7a77-be12-4292-a8ee-527af7270885_isr_945869_50.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 705e329b8d4bfc70ef3863c7a1381c7f with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 0bba0523eaafa9be7c6f3519a5348fca parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463911c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bd4b635b-1f03-4133-a660-8008a7656dff_characterizeImage_945869_36.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bd4b635b-1f03-4133-a660-8008a7656dff_characterizeImage_945869_36.stderr parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/77ad538e-3061-4a5a-a0a9-42ff4f1d1f12_isr_945869_36.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/77ad538e-3061-4a5a-a0a9-42ff4f1d1f12_isr_945869_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:48 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82774017d228b4d4fbffd4526ead425c with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 304cc564e66d4c08aa0902d3121f143a parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/7def5224-18a5-45a0-9a81-4a3ef64ef58b_characterizeImage_945755_58.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/7def5224-18a5-45a0-9a81-4a3ef64ef58b_characterizeImage_945755_58.stderr parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/a95707ef-0303-4975-8e84-60395aff6159_isr_945755_58.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/a95707ef-0303-4975-8e84-60395aff6159_isr_945755_58.stderr parsl.dataflow.dflow INFO: Task 960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05ee0aec8794a7408d7f510cfd5ba012 with result from task 960 parsl.dataflow.dflow DEBUG: Task 960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 961 has memoization hash 60f0617e669ac4ccefd68e17f9720f52 parsl.dataflow.memoization INFO: Task 961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/74e72023-1221-4174-bcd3-976947e3c1c1_characterizeImage_945867_24.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/74e72023-1221-4174-bcd3-976947e3c1c1_characterizeImage_945867_24.stderr parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/185ef91a-067c-4e62-b297-c79867b88bcd_isr_945867_24.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/185ef91a-067c-4e62-b297-c79867b88bcd_isr_945867_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:48 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:48 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4fb1d494af3ae740e312a8fe9b0ea68 with result from task 909 parsl.dataflow.dflow DEBUG: Task 909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash 3644f81ee40815484d53567637bfae60 parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/d142a2d6-48f3-4611-9dcf-338ff8834e5b_characterizeImage_945869_2.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/d142a2d6-48f3-4611-9dcf-338ff8834e5b_characterizeImage_945869_2.stderr parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/7d32b846-4754-43c6-8eae-ea04c36e4014_isr_945869_2.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/7d32b846-4754-43c6-8eae-ea04c36e4014_isr_945869_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:48 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d73761d29267f9e4778f4887113a7727 with result from task 840 parsl.dataflow.dflow DEBUG: Task 840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 841 has memoization hash 85302bfe79066072c5dc2251dc154cca parsl.dataflow.memoization INFO: Task 841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/567a953f-783e-437c-9a54-696bb38bf863_characterizeImage_945867_54.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/567a953f-783e-437c-9a54-696bb38bf863_characterizeImage_945867_54.stderr parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1eb5a2fc-b422-4ca2-8c58-66571a1d28e1_isr_945867_54.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1eb5a2fc-b422-4ca2-8c58-66571a1d28e1_isr_945867_54.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96d324796d286a32f720bc92d732948d with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash a9114dace4b67ef82345c0b47403a72d parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246390860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/133ce4d9-f333-4bb5-ba7e-e0b15a8bb2d2_characterizeImage_945869_54.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/133ce4d9-f333-4bb5-ba7e-e0b15a8bb2d2_characterizeImage_945869_54.stderr parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/c5140c1e-14fa-43be-810b-fdd3d7c73bd6_isr_945869_54.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/c5140c1e-14fa-43be-810b-fdd3d7c73bd6_isr_945869_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:49 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:49 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:49 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2d58fbbf417b1f38316471fdb76c09b with result from task 966 parsl.dataflow.dflow DEBUG: Task 966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 967 has memoization hash 1c4a665b7debfb58265e475cf61de76d parsl.dataflow.memoization INFO: Task 967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/5a242c61-4811-4646-82cb-0b774cc0bac0_characterizeImage_945867_6.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/5a242c61-4811-4646-82cb-0b774cc0bac0_characterizeImage_945867_6.stderr parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/dbcfa76d-6b4a-4f74-ab98-ec18f0def311_isr_945867_6.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/dbcfa76d-6b4a-4f74-ab98-ec18f0def311_isr_945867_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:49 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:49 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:50 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:50 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:50 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73c7482c88f61f81a7248546389adf84 with result from task 972 parsl.dataflow.dflow DEBUG: Task 972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 973 has memoization hash 53f0b170682269a7a8336ea2d7d5d994 parsl.dataflow.memoization INFO: Task 973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/fafae265-a6dc-4be3-9b51-110ea2e42065_characterizeImage_945869_14.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/fafae265-a6dc-4be3-9b51-110ea2e42065_characterizeImage_945869_14.stderr parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e92f2df4-1232-46bc-8cdb-1c4a679dc0d4_isr_945869_14.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e92f2df4-1232-46bc-8cdb-1c4a679dc0d4_isr_945869_14.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 869745d9e92ce4198c55ecbc1faedcc4 with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash 619f0f9310936fcb5485ee1bf5ed185b parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/48732780-b30b-4b32-8fba-d19a9a6f2ccd_characterizeImage_945756_33.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/48732780-b30b-4b32-8fba-d19a9a6f2ccd_characterizeImage_945756_33.stderr parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/17308fdb-e39e-47a7-afc7-2ac0e2959e8f_isr_945756_33.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/17308fdb-e39e-47a7-afc7-2ac0e2959e8f_isr_945756_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:50 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 862e2baf268b106e0f4af84ad9e833e6 with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash a24f5dda747590be9d7282714b2e9f57 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/57ff429e-d686-4475-95eb-c88b72f967bc_characterizeImage_945867_32.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/57ff429e-d686-4475-95eb-c88b72f967bc_characterizeImage_945867_32.stderr parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/c7348a38-4dd3-47a7-bb3e-91ae8c5f8e52_isr_945867_32.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/c7348a38-4dd3-47a7-bb3e-91ae8c5f8e52_isr_945867_32.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2f042f53a539e883fef9cf07e6e665f with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash 92ca9d8a864a32d515e8e27267e4a287 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/5317748c-b718-4dc7-acfc-e900da9b12d1_characterizeImage_945757_33.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/5317748c-b718-4dc7-acfc-e900da9b12d1_characterizeImage_945757_33.stderr parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/ba4bb611-8169-4edb-a78f-37d6fc108caf_isr_945757_33.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/ba4bb611-8169-4edb-a78f-37d6fc108caf_isr_945757_33.stderr parsl.dataflow.dflow INFO: Task 1095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e212f6ab7c79886bd348ca803fbcb7c6 with result from task 1095 parsl.dataflow.dflow DEBUG: Task 1095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash 528fee933666b9e80d16285f4e960cfd parsl.dataflow.memoization INFO: Task 1096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1096 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/cef9edf2-6788-410a-ab77-f0843b946e4c_characterizeImage_945867_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/cef9edf2-6788-410a-ab77-f0843b946e4c_characterizeImage_945867_3.stderr parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/c12dbaf2-b061-4020-8399-7415662b6e9f_isr_945867_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/c12dbaf2-b061-4020-8399-7415662b6e9f_isr_945867_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:51 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:51 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:51 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86150fa544be0f7ec9d63364e5c58578 with result from task 969 parsl.dataflow.dflow DEBUG: Task 969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 970 has memoization hash 103f62c0a03ca52a1be27928f7801fc9 parsl.dataflow.memoization INFO: Task 970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/eb5af35a-aa91-4585-ab54-68f4edfe7947_characterizeImage_945756_17.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/eb5af35a-aa91-4585-ab54-68f4edfe7947_characterizeImage_945756_17.stderr parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/61e3df39-a954-451c-b8d4-8c14e0bc85f3_isr_945756_17.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/61e3df39-a954-451c-b8d4-8c14e0bc85f3_isr_945756_17.stderr parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfea82a6cf5f9ab404d7ef0c5b2a990f with result from task 843 parsl.dataflow.dflow DEBUG: Task 843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 844 has memoization hash d8ed1f3c310ffb1d4a27b270beca359e parsl.dataflow.memoization INFO: Task 844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/7972cfd9-c795-46bc-8414-bde02cceefa3_characterizeImage_945756_49.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/7972cfd9-c795-46bc-8414-bde02cceefa3_characterizeImage_945756_49.stderr parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/71632fe1-8200-4813-a47c-5d3d6aee2d44_isr_945756_49.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/71632fe1-8200-4813-a47c-5d3d6aee2d44_isr_945756_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:52 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:52 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:52 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3627e23f063da9cf8aba46e47f004daf with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 4b01dabfc20f75f8cd8df39f7736f3f8 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/1bb69e89-133f-408d-93bb-5f932a70dd90_characterizeImage_945867_38.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/1bb69e89-133f-408d-93bb-5f932a70dd90_characterizeImage_945867_38.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/127df8ec-3426-4d59-8c94-c0eb60c6b618_isr_945867_38.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/127df8ec-3426-4d59-8c94-c0eb60c6b618_isr_945867_38.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23b52474f0947c5b8757c8941fdd6fe3 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 14b98f19e2da1bea94d95b40c263fa97 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/8596220f-13ef-4619-8160-dba51ba73132_characterizeImage_945755_42.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/8596220f-13ef-4619-8160-dba51ba73132_characterizeImage_945755_42.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/84cb85c2-c72f-4e2a-b874-502221962f5f_isr_945755_42.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/84cb85c2-c72f-4e2a-b874-502221962f5f_isr_945755_42.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12fc8abc15cccab5ceb9fc745663706e with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 6c175cab1b26e96c244a5dda892aed88 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/2507f02a-bad6-4b16-8547-727a833ce23a_characterizeImage_945755_59.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/2507f02a-bad6-4b16-8547-727a833ce23a_characterizeImage_945755_59.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8df7866a-3a82-4662-b9ac-f170cc16ada2_isr_945755_59.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8df7866a-3a82-4662-b9ac-f170cc16ada2_isr_945755_59.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f76fb3d6f11cf2ec6aec8b9c823e36b2 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 336da3fc0001dee0724aed037b4a3591 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/c86ca45e-8a06-4332-a812-96dc36252dda_characterizeImage_945757_45.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/c86ca45e-8a06-4332-a812-96dc36252dda_characterizeImage_945757_45.stderr parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/1a229376-2bea-43e6-b91b-2aaae18f39a5_isr_945757_45.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/1a229376-2bea-43e6-b91b-2aaae18f39a5_isr_945757_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:52 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:52 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:53 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:53 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d468d2f563c27544bad8f20f0592445 with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 00d284721d55a4292200943ce077e699 parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/a9a29306-aa2a-4a76-847d-f05602234e2c_characterizeImage_945869_13.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/a9a29306-aa2a-4a76-847d-f05602234e2c_characterizeImage_945869_13.stderr parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/3abdb6f9-03ad-44a6-a5e0-f7f5f46b1d88_isr_945869_13.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/3abdb6f9-03ad-44a6-a5e0-f7f5f46b1d88_isr_945869_13.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f91e5d79dddf1ca871aeedfc74357252 with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash ce94fe54b741fabcf12a2e3fb940ceb7 parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/0e2987d5-26a8-4816-bbf1-86e74daf5871_characterizeImage_945867_33.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/0e2987d5-26a8-4816-bbf1-86e74daf5871_characterizeImage_945867_33.stderr parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8b4b0938-357d-4d01-9d15-618d45da728c_isr_945867_33.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8b4b0938-357d-4d01-9d15-618d45da728c_isr_945867_33.stderr parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8e0a2549ceb80715d2f4503b652225c with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 5987e08fa443c826630346433877043c parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/1b3f9f7f-22a6-4180-bb3f-be829358a380_characterizeImage_945868_32.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/1b3f9f7f-22a6-4180-bb3f-be829358a380_characterizeImage_945868_32.stderr parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/829476d6-d576-43ea-b12e-9fc1d8f51348_isr_945868_32.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/829476d6-d576-43ea-b12e-9fc1d8f51348_isr_945868_32.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd2c173b8e1e301851522316f5052908 with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 0aa2016817e3d61087955d34bcf9f425 parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/2ff24112-6dcf-42a6-a3f5-71155d5ab7f3_characterizeImage_945868_8.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/2ff24112-6dcf-42a6-a3f5-71155d5ab7f3_characterizeImage_945868_8.stderr parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/72bf8367-6ac4-408c-8512-29ed514b9632_isr_945868_8.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/72bf8367-6ac4-408c-8512-29ed514b9632_isr_945868_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:53 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:53 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cebd7a0da11d12e249302cdc9ab2e68b with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash d909fa9b08d3835a41a437cc8d511f81 parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/238d3a32-e75c-46e9-9c8e-373696c7dc25_characterizeImage_945756_15.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/238d3a32-e75c-46e9-9c8e-373696c7dc25_characterizeImage_945756_15.stderr parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/186a4c82-d3bd-4f41-a392-7423820519ca_isr_945756_15.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/186a4c82-d3bd-4f41-a392-7423820519ca_isr_945756_15.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d1408991b94d01d6a3bdce6cd146734 with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 28010ef00dbc55a1eec319b8eb0efdbd parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:53 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/89b844e5-3d1d-4075-b19c-ab6fe2603061_characterizeImage_945867_60.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/89b844e5-3d1d-4075-b19c-ab6fe2603061_characterizeImage_945867_60.stderr parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4e507d82-3fbd-47fc-b365-29418154e3bb_isr_945867_60.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4e507d82-3fbd-47fc-b365-29418154e3bb_isr_945867_60.stderr parsl.dataflow.dflow INFO: Task 1098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ca88035720fd2b222776fe8d09415e7 with result from task 1098 parsl.dataflow.dflow DEBUG: Task 1098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1099 has memoization hash f0b1603535d57500270d224be3be964f parsl.dataflow.memoization INFO: Task 1099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1099 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b79a326e-66a4-4ab8-a43e-e568e6ba0fef_characterizeImage_945757_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b79a326e-66a4-4ab8-a43e-e568e6ba0fef_characterizeImage_945757_53.stderr parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/527557a9-b9a4-4332-b751-523e2f4e962b_isr_945757_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/527557a9-b9a4-4332-b751-523e2f4e962b_isr_945757_53.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6045028373dca2951a4c0444732778b9 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash ce01ef13938671b33bd29c1990df020f parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/896517ee-ccbf-4c0f-80c2-058aea885946_characterizeImage_945867_12.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/896517ee-ccbf-4c0f-80c2-058aea885946_characterizeImage_945867_12.stderr parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8a936f05-2edd-4128-89fe-1720f61c3a4d_isr_945867_12.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8a936f05-2edd-4128-89fe-1720f61c3a4d_isr_945867_12.stderr parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 137722b0c913efb8887fd38a2e40b9a6 with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash a4c82cba98e1ac8ca1acb029355d8474 parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a6e43120-948b-4fba-9cbc-fdd58d567c5c_characterizeImage_945757_19.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a6e43120-948b-4fba-9cbc-fdd58d567c5c_characterizeImage_945757_19.stderr parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f7aa67c1-b06d-470d-ab69-020a5ed2e3e5_isr_945757_19.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f7aa67c1-b06d-470d-ab69-020a5ed2e3e5_isr_945757_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:54 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1792032cfb4806661963d6709db55be3 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash aea6fb08df13120c477fe4bf6aae5ee0 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/2077d2a9-3049-4890-aaee-6c24952558d2_characterizeImage_945756_54.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/2077d2a9-3049-4890-aaee-6c24952558d2_characterizeImage_945756_54.stderr parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/27d7efcc-4123-41f9-8c71-2dcdb092d103_isr_945756_54.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/27d7efcc-4123-41f9-8c71-2dcdb092d103_isr_945756_54.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4447c39c356619865263f6eb861dd81b with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash f7d88b81a308147f337c539dbff027b0 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2dea0aa4-1dec-4060-9374-c19868103550_characterizeImage_945867_58.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2dea0aa4-1dec-4060-9374-c19868103550_characterizeImage_945867_58.stderr parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/144763f3-c7f2-4def-a169-7d14727eb42d_isr_945867_58.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/144763f3-c7f2-4def-a169-7d14727eb42d_isr_945867_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:54 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de854ddb39582f910f6963c7b9b7e866 with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 82a09c7a821b4dc4d9f4cc49c3aade2e parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4830841d-90f4-4bfa-a89e-74196b499a38_characterizeImage_945867_13.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4830841d-90f4-4bfa-a89e-74196b499a38_characterizeImage_945867_13.stderr parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/f8b7576c-d600-48c1-b0ac-f3ac20af6c7a_isr_945867_13.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/f8b7576c-d600-48c1-b0ac-f3ac20af6c7a_isr_945867_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:54 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70cd231845cb2ab6f7ebeb7e582486bb with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash 4d37b347ba0d34e9722a61e86b921f8a parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/714f1228-0972-4bd0-b39b-8581c6b8d9a0_characterizeImage_945869_55.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/714f1228-0972-4bd0-b39b-8581c6b8d9a0_characterizeImage_945869_55.stderr parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8c845107-2787-4ced-bb15-08fa1d0fec2d_isr_945869_55.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8c845107-2787-4ced-bb15-08fa1d0fec2d_isr_945869_55.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cdd1f5802d975c7e3221808ba58bf74 with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 8bd56bb684f1fa48f82c9a25a6901a17 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246390b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/6d35d883-8bd0-4744-9a2f-9dea1ce6935c_characterizeImage_945757_57.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/6d35d883-8bd0-4744-9a2f-9dea1ce6935c_characterizeImage_945757_57.stderr parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/fd243099-165c-430f-aff4-87d1e3d88941_isr_945757_57.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/fd243099-165c-430f-aff4-87d1e3d88941_isr_945757_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:55 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c29d7900293f1c44604c74c76e2efc1b with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 529ff15e9de4e1451fa2be6c465c1b1e parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/8d025b50-a8ec-48a9-886f-5f55b1b528a9_characterizeImage_945869_57.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/8d025b50-a8ec-48a9-886f-5f55b1b528a9_characterizeImage_945869_57.stderr parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/87f34727-610d-4d86-89ac-84359660e677_isr_945869_57.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/87f34727-610d-4d86-89ac-84359660e677_isr_945869_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:55 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00d7dc005bb2d604325426fc5fd6bdcf with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 6cbeaa09eb8cd7d2fbae862376ae7c2f parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462be0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/e27ccb4e-4b6e-44fc-9467-fe4f9663bbb2_characterizeImage_945868_20.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/e27ccb4e-4b6e-44fc-9467-fe4f9663bbb2_characterizeImage_945868_20.stderr parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/b1c55525-4144-41a9-82df-6d2fd9e9672f_isr_945868_20.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/b1c55525-4144-41a9-82df-6d2fd9e9672f_isr_945868_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:55 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87013e20880a79cf5f06607d59951757 with result from task 852 parsl.dataflow.dflow DEBUG: Task 852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 853 has memoization hash f1347c9ffc7c2b0435e29dcebc02ee70 parsl.dataflow.memoization INFO: Task 853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/066c3b35-648c-40ea-b979-ef2ba521535f_characterizeImage_945756_13.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/066c3b35-648c-40ea-b979-ef2ba521535f_characterizeImage_945756_13.stderr parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/7aedc253-37c9-47a8-8dc1-2bcaf2a9e0b1_isr_945756_13.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/7aedc253-37c9-47a8-8dc1-2bcaf2a9e0b1_isr_945756_13.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 402016ac0f8a6f37023440b72d0008f3 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 45db7ac2a82f88bd80ff37bac26a2d4a parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/109edcee-de71-4665-a0bb-9c8ade9f452d_characterizeImage_945756_61.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/109edcee-de71-4665-a0bb-9c8ade9f452d_characterizeImage_945756_61.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/03432321-7b36-4944-a3f4-306cb2729efa_isr_945756_61.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/03432321-7b36-4944-a3f4-306cb2729efa_isr_945756_61.stderr parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8d7c91f8c98a56454dec52f6891be3e with result from task 849 parsl.dataflow.dflow DEBUG: Task 849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 850 has memoization hash 41ab9ee151428c7254623837c200ae1b parsl.dataflow.memoization INFO: Task 850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2d68eb6e-18ce-4d92-88dc-cf73e7266c33_characterizeImage_945869_58.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2d68eb6e-18ce-4d92-88dc-cf73e7266c33_characterizeImage_945869_58.stderr parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/a4a732c0-8aaa-4449-963d-8c4e485b8392_isr_945869_58.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/a4a732c0-8aaa-4449-963d-8c4e485b8392_isr_945869_58.stderr parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f7f8f8c49013b31c81416eea90777b2 with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash aec8578e6892ad2543edbc3386397059 parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/7f14d69f-accb-4be7-a3bd-cca87839e1a3_characterizeImage_945867_10.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/7f14d69f-accb-4be7-a3bd-cca87839e1a3_characterizeImage_945867_10.stderr parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/f1960032-e7fe-409d-a319-a17103452f72_isr_945867_10.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/f1960032-e7fe-409d-a319-a17103452f72_isr_945867_10.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d112bc92a1189e017590db2f41b22db with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 7448c1a430eb7ba7cbe51bd5dac99cf4 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1aa08685-a257-47e2-9b48-f866df66125f_characterizeImage_945869_12.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1aa08685-a257-47e2-9b48-f866df66125f_characterizeImage_945869_12.stderr parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e990570a-3981-4d31-80f0-49763a857108_isr_945869_12.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e990570a-3981-4d31-80f0-49763a857108_isr_945869_12.stderr parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52f585708c2fc6a5366109278b25952a with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash aff0bc1a4011f1c01f935d1c59bb8981 parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/868de034-4bc3-41a0-8d14-98a1a6c6555e_characterizeImage_945867_39.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/868de034-4bc3-41a0-8d14-98a1a6c6555e_characterizeImage_945867_39.stderr parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/69067d16-d94c-4d38-a412-8e6f98c660f8_isr_945867_39.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/69067d16-d94c-4d38-a412-8e6f98c660f8_isr_945867_39.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a1ad88fbc89b82f6bb1e817ef3e51f6 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash e7a64722431f58b191a4205e10202aca parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/9cd307bc-e425-4510-8c06-01524416f8cd_characterizeImage_945755_41.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/9cd307bc-e425-4510-8c06-01524416f8cd_characterizeImage_945755_41.stderr parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/31716c3e-36bf-4de2-85a5-508d49e24aca_isr_945755_41.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/31716c3e-36bf-4de2-85a5-508d49e24aca_isr_945755_41.stderr parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9550b820cfd7945895d771f29ed6a3f0 with result from task 855 parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:56 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow DEBUG: Task 855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 856 has memoization hash bd54833672d495ef22bfb0b47b27eded parsl.dataflow.memoization INFO: Task 856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aa020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/a816fa1a-f416-4e44-92ba-02c024a0d1bb_characterizeImage_945755_14.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/a816fa1a-f416-4e44-92ba-02c024a0d1bb_characterizeImage_945755_14.stderr parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/6228746c-7bdf-4458-a30c-f25ae39eab3c_isr_945755_14.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/6228746c-7bdf-4458-a30c-f25ae39eab3c_isr_945755_14.stderr parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96eb6c1c6c80984e3b4bd23d0529566b with result from task 846 parsl.dataflow.dflow DEBUG: Task 846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 847 has memoization hash 39aa50c0686a56bace3f68ebb01160d3 parsl.dataflow.memoization INFO: Task 847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/4ca5bb72-b1b5-49f2-a4aa-f4c8aa6e55d0_characterizeImage_945869_47.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/4ca5bb72-b1b5-49f2-a4aa-f4c8aa6e55d0_characterizeImage_945869_47.stderr parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f79b92e5-0d92-43de-90fc-56e87cb196d4_isr_945869_47.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f79b92e5-0d92-43de-90fc-56e87cb196d4_isr_945869_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:56 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c57e86bd6c070ea2c8a383a2683052cc with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 9e17887fb5d6f08cc166e971d1cc7359 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f472cb4b-8811-4352-a411-3ba2e52087b9_characterizeImage_945756_62.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f472cb4b-8811-4352-a411-3ba2e52087b9_characterizeImage_945756_62.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/9ee6bfd5-475d-43e4-94fb-f35baff641cb_isr_945756_62.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/9ee6bfd5-475d-43e4-94fb-f35baff641cb_isr_945756_62.stderr parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2d2008ac36b28f6a0d051ce1e8d56d7 with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 9c7b0a483f6227edf2b40b4d5166b9d4 parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/0dff9db8-cb1f-4975-ac61-3b1093211a43_characterizeImage_945755_45.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/0dff9db8-cb1f-4975-ac61-3b1093211a43_characterizeImage_945755_45.stderr parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/96784ad5-e339-43c2-93f1-2a78614f5666_isr_945755_45.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/96784ad5-e339-43c2-93f1-2a78614f5666_isr_945755_45.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69099a3238d916383fa0c2232ca62c6e with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash a86c11eaa2c77a502e435a2d777c8cdd parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/3e50b388-2dfa-4a98-aae7-01d790c4085a_characterizeImage_945868_38.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/3e50b388-2dfa-4a98-aae7-01d790c4085a_characterizeImage_945868_38.stderr parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/a63c1ac2-f9aa-40b8-8f70-f1a48c0d6eb5_isr_945868_38.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/a63c1ac2-f9aa-40b8-8f70-f1a48c0d6eb5_isr_945868_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:57 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:57 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:58 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f5130620304c94be52acc89c7c18a77 with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash 73c6972eefb820e04974f6089dfe730b parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/6b69b5fc-ed72-47a4-8b3e-8346d51be56d_characterizeImage_945757_38.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/6b69b5fc-ed72-47a4-8b3e-8346d51be56d_characterizeImage_945757_38.stderr parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/ccfc3bcb-be79-4663-968d-183d61a5794b_isr_945757_38.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/ccfc3bcb-be79-4663-968d-183d61a5794b_isr_945757_38.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98e274d58b0e8c522ca50fd9cb750338 with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash a51c05554f557bf44f3ba3a61c6b6991 parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/929a82bc-ab10-4c8a-ab66-3caca7ae17dd_characterizeImage_945755_5.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/929a82bc-ab10-4c8a-ab66-3caca7ae17dd_characterizeImage_945755_5.stderr parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/636ef77a-6355-462b-8b8c-d6721212e316_isr_945755_5.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/636ef77a-6355-462b-8b8c-d6721212e316_isr_945755_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:58 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:34:59 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31c5a23015feb8af9136ec1a57ff9bf3 with result from task 822 parsl.dataflow.dflow DEBUG: Task 822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 823 has memoization hash 0627a026ed7c5bf155c73fdbb7133508 parsl.dataflow.memoization INFO: Task 823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/69f7205a-a3b6-4d1d-acf4-36fcf6d6a549_characterizeImage_945868_13.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/69f7205a-a3b6-4d1d-acf4-36fcf6d6a549_characterizeImage_945868_13.stderr parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d3ad9281-63af-4024-8331-5a38c7cb8c81_isr_945868_13.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d3ad9281-63af-4024-8331-5a38c7cb8c81_isr_945868_13.stderr parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2613893783cc22a635973069167ea092 with result from task 825 parsl.dataflow.dflow DEBUG: Task 825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 826 has memoization hash 8f5a6708aa28b2554b295cf17a0e527f parsl.dataflow.memoization INFO: Task 826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a15981f6-22e2-41b1-9c41-dfc7a2f76fdb_characterizeImage_945868_46.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a15981f6-22e2-41b1-9c41-dfc7a2f76fdb_characterizeImage_945868_46.stderr parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/2de1d611-3964-4550-be33-249de877734a_isr_945868_46.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/2de1d611-3964-4550-be33-249de877734a_isr_945868_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:00 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af60e4431dea480e4724dfdf56a142db with result from task 837 parsl.dataflow.dflow DEBUG: Task 837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 838 has memoization hash 76d17711eecb491969034d285c489294 parsl.dataflow.memoization INFO: Task 838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/c9b20239-a926-41d1-bf32-582de2e5334b_characterizeImage_945868_48.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/c9b20239-a926-41d1-bf32-582de2e5334b_characterizeImage_945868_48.stderr parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/01ee1bf3-75e2-49e7-914b-105f2b48c6ba_isr_945868_48.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/01ee1bf3-75e2-49e7-914b-105f2b48c6ba_isr_945868_48.stderr parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1302c45c6c75a0f6f60356b97e929a73 with result from task 816 parsl.dataflow.dflow DEBUG: Task 816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 817 has memoization hash 0be21ad542cc6e0bf6ffbb8b23dc4d12 parsl.dataflow.memoization INFO: Task 817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/91698a2d-4208-4544-af73-5615a7fc30dc_characterizeImage_945868_31.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/91698a2d-4208-4544-af73-5615a7fc30dc_characterizeImage_945868_31.stderr parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/e984bfc5-ad5f-4fb7-ad5c-835040bba7cf_isr_945868_31.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/e984bfc5-ad5f-4fb7-ad5c-835040bba7cf_isr_945868_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:00 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffb5cc0c78919b531f651029985236ff with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash a6396d82ddfb35c0d279a4083a163ccf parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ba0808e8-c461-4f0d-8b4f-b22c688e534a_characterizeImage_945756_53.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ba0808e8-c461-4f0d-8b4f-b22c688e534a_characterizeImage_945756_53.stderr parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/5e64e9f1-0a44-4d70-830b-87a530baf0f8_isr_945756_53.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/5e64e9f1-0a44-4d70-830b-87a530baf0f8_isr_945756_53.stderr parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a045305d746f82a21acc14b2737f5afb with result from task 831 parsl.dataflow.dflow DEBUG: Task 831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 832 has memoization hash 83f2fe949445babcb01a601699a95073 parsl.dataflow.memoization INFO: Task 832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/814c29b8-cb7d-40dc-ab68-a71f77986dcf_characterizeImage_945757_18.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/814c29b8-cb7d-40dc-ab68-a71f77986dcf_characterizeImage_945757_18.stderr parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/fc11d2d4-8971-4a72-90a4-d0a3d2b85221_isr_945757_18.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/fc11d2d4-8971-4a72-90a4-d0a3d2b85221_isr_945757_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:01 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:01 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a4e3a58302793befd95dbec10348582 with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash 10aa0edd61cab12087f016e97c5e738a parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/7f0d345f-e995-41ca-82e2-6476868dc21b_characterizeImage_945755_27.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/7f0d345f-e995-41ca-82e2-6476868dc21b_characterizeImage_945755_27.stderr parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/58922fbf-1fff-4db5-b7d1-bab2351756f9_isr_945755_27.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/58922fbf-1fff-4db5-b7d1-bab2351756f9_isr_945755_27.stderr parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bed32f4702971ee663955f18094fa8d7 with result from task 834 parsl.dataflow.dflow DEBUG: Task 834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 835 has memoization hash 8b94d75c73cf34594405115048141f32 parsl.dataflow.memoization INFO: Task 835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/8a1cd2a5-b287-41af-bb27-577a4e47454d_characterizeImage_945867_62.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/8a1cd2a5-b287-41af-bb27-577a4e47454d_characterizeImage_945867_62.stderr parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/dcba9cf3-7879-4f45-adc2-33c6c07f2b92_isr_945867_62.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/dcba9cf3-7879-4f45-adc2-33c6c07f2b92_isr_945867_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:02 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:02 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e44ee332c01c6a921c3619f39f483746 with result from task 828 parsl.dataflow.dflow DEBUG: Task 828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 829 has memoization hash 2f90f5db473d2453d8f274e1ec86a499 parsl.dataflow.memoization INFO: Task 829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/c9f57031-214e-41ec-bae6-053386243caf_characterizeImage_945869_37.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/c9f57031-214e-41ec-bae6-053386243caf_characterizeImage_945869_37.stderr parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/20f2cfc7-07c8-435b-96de-d6d69219430a_isr_945869_37.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/20f2cfc7-07c8-435b-96de-d6d69219430a_isr_945869_37.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 695616e333c0a6f6feb3b24f9caaa100 with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash 3d4211db9d41185c8709b2b6b08007ef parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/621c1adf-bf2f-4676-83de-a3ddcccee9b2_characterizeImage_945869_32.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/621c1adf-bf2f-4676-83de-a3ddcccee9b2_characterizeImage_945869_32.stderr parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/9ec1ef08-3848-481b-a922-d0eaf7006bc4_isr_945869_32.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/9ec1ef08-3848-481b-a922-d0eaf7006bc4_isr_945869_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:03 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b565df00186891eb1403a0a58ca49927 with result from task 819 parsl.dataflow.dflow DEBUG: Task 819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 820 has memoization hash 877d0b8a3d75e81d2013d53f2448ec25 parsl.dataflow.memoization INFO: Task 820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ae7efdd8-b351-4590-a9c8-07879a426918_characterizeImage_945756_10.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ae7efdd8-b351-4590-a9c8-07879a426918_characterizeImage_945756_10.stderr parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/97f6c2de-1d47-47d6-b5d0-5d73a233c6f2_isr_945756_10.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/97f6c2de-1d47-47d6-b5d0-5d73a233c6f2_isr_945756_10.stderr parsl.dataflow.dflow INFO: Task 1101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f65339178332198ac6c79df315746f3 with result from task 1101 parsl.dataflow.dflow DEBUG: Task 1101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1102 has memoization hash 8049fa7d3679e690174992f6037ede1b parsl.dataflow.memoization INFO: Task 1102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1102 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ff5a127a-dda7-4da1-949d-674691b99557_characterizeImage_945756_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ff5a127a-dda7-4da1-949d-674691b99557_characterizeImage_945756_28.stderr parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/f827be57-aa3c-49fc-b3a9-b1577262d7b6_isr_945756_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/f827be57-aa3c-49fc-b3a9-b1577262d7b6_isr_945756_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:04 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a223f80d132423025958c3272bb52f44 with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash 943fa85287bffde6237d183e4aa4842e parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/21a84baa-2f5c-474a-9df6-d555b46996a5_characterizeImage_945757_46.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/21a84baa-2f5c-474a-9df6-d555b46996a5_characterizeImage_945757_46.stderr parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f0f932a9-5280-41b9-91e6-699b77217ec7_isr_945757_46.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/f0f932a9-5280-41b9-91e6-699b77217ec7_isr_945757_46.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 49 try 0 failed parsl.dataflow.dflow INFO: Task 49 marked for retry parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/699acbed-ec41-4e6a-a585-6a294ce9b24f_characterizeImage_945755_2.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/699acbed-ec41-4e6a-a585-6a294ce9b24f_characterizeImage_945755_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 9b296c3265ac177d46ab3db73e164dd7 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 1 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/699acbed-ec41-4e6a-a585-6a294ce9b24f_characterizeImage_945755_2.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/699acbed-ec41-4e6a-a585-6a294ce9b24f_characterizeImage_945755_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:04 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb3a65b1f6c6fccb27f014e2c3a23a3b with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 3caece5066cdee40f257e36a6dceaaaf parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/d11060d0-735f-4c3d-b252-5d82c4de982a_characterizeImage_945755_20.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/d11060d0-735f-4c3d-b252-5d82c4de982a_characterizeImage_945755_20.stderr parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3eb21e85-30ab-4586-a8d6-4f4abea41e1a_isr_945755_20.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3eb21e85-30ab-4586-a8d6-4f4abea41e1a_isr_945755_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:05 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07049c6b6fa9575b0922a8a9ea80d8ff with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 60d7a3809bdcac614c4a210af4a200cc parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4808126a-c807-41ab-8871-bc476ef84391_characterizeImage_945867_8.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4808126a-c807-41ab-8871-bc476ef84391_characterizeImage_945867_8.stderr parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/cff5b11a-a171-4fc5-8755-d3e63d70140b_isr_945867_8.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/cff5b11a-a171-4fc5-8755-d3e63d70140b_isr_945867_8.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7df946c0265cee6b357f163af3c4bcd8 with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash c25adc0c288e1860489bc59e7a380d97 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/cfcb946f-09cb-41fd-81ac-b54e8f64e336_characterizeImage_945868_51.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/cfcb946f-09cb-41fd-81ac-b54e8f64e336_characterizeImage_945868_51.stderr parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/87bada8d-8e07-4292-9985-c16920ed3dce_isr_945868_51.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/87bada8d-8e07-4292-9985-c16920ed3dce_isr_945868_51.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05ddd63125af1d9a46c95f3bc9f1278b with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 28136ca1001ca174e931d3c45fb4d77b parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b4dbbe53-9382-4a64-8d0e-0dd101dfc7e3_characterizeImage_945756_5.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b4dbbe53-9382-4a64-8d0e-0dd101dfc7e3_characterizeImage_945756_5.stderr parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0cf2a0db-9575-49bc-a01f-2bb29683cc96_isr_945756_5.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0cf2a0db-9575-49bc-a01f-2bb29683cc96_isr_945756_5.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 144b2844c4add818fefb5ff5115a1d38 with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 383284de431821370c02ffdf2d90381a parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462be700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/764407ce-23da-460f-83c7-43d5e8ba9f5f_characterizeImage_945868_40.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/764407ce-23da-460f-83c7-43d5e8ba9f5f_characterizeImage_945868_40.stderr parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/a219b5d4-d320-4cef-8055-873221230d17_isr_945868_40.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/a219b5d4-d320-4cef-8055-873221230d17_isr_945868_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:05 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd51c2c7d472fdcaea3282c9117f2460 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 86b5c07678188607a4e31d06c61e769f parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/6876be82-4175-426f-bca0-ccb3b08f9f12_characterizeImage_945869_24.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/6876be82-4175-426f-bca0-ccb3b08f9f12_characterizeImage_945869_24.stderr parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f0bb07d6-28af-4a88-8ccb-b0578fc56e05_isr_945869_24.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f0bb07d6-28af-4a88-8ccb-b0578fc56e05_isr_945869_24.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e800d068bcd0d980c8639d5aa9f154f6 with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 31b9e0f5ed720d8ea6008a45cd137a4f parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/dd5e5de6-f674-4331-b315-5d634dac905f_characterizeImage_945755_4.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/dd5e5de6-f674-4331-b315-5d634dac905f_characterizeImage_945755_4.stderr parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/7892dd7e-24f0-4007-9426-87c172610a3d_isr_945755_4.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/7892dd7e-24f0-4007-9426-87c172610a3d_isr_945755_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:06 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62f5df575e3d18c3ff6480a740cb8fd4 with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 53205458b27a23665e62929cc1eb0c74 parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/cfb4d370-eebb-4383-8b10-05f7b47cf5ac_characterizeImage_945868_62.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/cfb4d370-eebb-4383-8b10-05f7b47cf5ac_characterizeImage_945868_62.stderr parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/94e83da6-1063-45f6-abd3-fd752121b4e9_isr_945868_62.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/94e83da6-1063-45f6-abd3-fd752121b4e9_isr_945868_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:06 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06ed5230df4dcc41f1ddbf4f604dac75 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 5c057a8ebafcb4b57509a61a11a40f46 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/caee9d8f-db79-48ca-9e09-98c915867ba1_characterizeImage_945868_60.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/caee9d8f-db79-48ca-9e09-98c915867ba1_characterizeImage_945868_60.stderr parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/96c83b62-ac0b-461f-9606-9474cf6ed92e_isr_945868_60.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/96c83b62-ac0b-461f-9606-9474cf6ed92e_isr_945868_60.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe41cd355ec554a2fcd1a5db54daf2af with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 3fa24061c98877392abb6fd68790ea1b parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ede4b03b-b547-415d-a8ed-f2b96aaa54b6_characterizeImage_945867_29.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ede4b03b-b547-415d-a8ed-f2b96aaa54b6_characterizeImage_945867_29.stderr parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/02464a48-7f01-48d8-b791-fac5800e03a4_isr_945867_29.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/02464a48-7f01-48d8-b791-fac5800e03a4_isr_945867_29.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1edf7f1c1dedbc22a5e21176c6d26d62 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash afa3eaa46d4b190425a412f2c775db87 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/f005b650-b0a4-4db2-a4b0-a0e9c1ff55b8_characterizeImage_945755_36.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/f005b650-b0a4-4db2-a4b0-a0e9c1ff55b8_characterizeImage_945755_36.stderr parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/1e29b5b3-a49d-48e7-9a84-067d5b0c02e8_isr_945755_36.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/1e29b5b3-a49d-48e7-9a84-067d5b0c02e8_isr_945755_36.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 640d4e235f340ecda40de4e1a761e0bf with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash b62cbdc9e9ed04f1e3d0778c07b57def parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/01809cc4-9ff1-4ce1-9493-b7ba4a91f98c_characterizeImage_945755_34.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/01809cc4-9ff1-4ce1-9493-b7ba4a91f98c_characterizeImage_945755_34.stderr parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/616cd863-39be-4738-9e2d-6e6615c99856_isr_945755_34.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/616cd863-39be-4738-9e2d-6e6615c99856_isr_945755_34.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fc1bdd2380685797830c4842b3a2960 with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 70bb369dbf4b7aa6d3e92d2f4c788af2 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/b5b60afa-c541-4296-90c4-17eaf426ae32_characterizeImage_945867_31.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/b5b60afa-c541-4296-90c4-17eaf426ae32_characterizeImage_945867_31.stderr parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/41fa5242-4fac-48a6-8b97-aae13d5654f4_isr_945867_31.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/41fa5242-4fac-48a6-8b97-aae13d5654f4_isr_945867_31.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e88cc7e5c6f5ce4b95007f858478b347 with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 4a0c8ef3a2c16834d5ce3997f5ea0d03 parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/493d2468-e857-4177-9299-99980dc497bf_characterizeImage_945867_18.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/493d2468-e857-4177-9299-99980dc497bf_characterizeImage_945867_18.stderr parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/41cdf9ba-bf9b-42b6-bac3-da11d15614c0_isr_945867_18.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/41cdf9ba-bf9b-42b6-bac3-da11d15614c0_isr_945867_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:07 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ad16655322fb26d19dfe54be1f28c1c with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 36f157d18b81f633ea5c9f9bc3476ee6 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/ddaa29be-f357-4a7d-91a7-ad9db322191f_characterizeImage_945868_6.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/ddaa29be-f357-4a7d-91a7-ad9db322191f_characterizeImage_945868_6.stderr parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9d786e71-b5a1-475a-82e3-e197d30b476f_isr_945868_6.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9d786e71-b5a1-475a-82e3-e197d30b476f_isr_945868_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:07 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3852bc7ff93815730df20f6cacd50a0f with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash bde5d3607e215eed193b9122ef1ae327 parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462be3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/ec019aa9-3ea6-4410-a5e6-1b7cf7a30ba8_characterizeImage_945869_31.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/ec019aa9-3ea6-4410-a5e6-1b7cf7a30ba8_characterizeImage_945869_31.stderr parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/b72091e8-354e-4c28-ab9b-ca0e905ff338_isr_945869_31.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/b72091e8-354e-4c28-ab9b-ca0e905ff338_isr_945869_31.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1f10890ad7be0a55ff62029f4ab0145 with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash a14e4610488697fb9d470f35439214c6 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/0e076001-03f3-44f8-a9ea-968a3299c4ca_characterizeImage_945756_9.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/0e076001-03f3-44f8-a9ea-968a3299c4ca_characterizeImage_945756_9.stderr parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/5c77a25f-dee2-4034-b206-7c389526f07e_isr_945756_9.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/5c77a25f-dee2-4034-b206-7c389526f07e_isr_945756_9.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6b544821ebbfd38952db9995ff56cf2 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 420df8fc323f804741ad24a9ab6b720f parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/2db9bd7d-28b7-4555-a566-04673aa088b5_characterizeImage_945868_7.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/2db9bd7d-28b7-4555-a566-04673aa088b5_characterizeImage_945868_7.stderr parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/7f7bee1f-14c7-42f8-b5a9-37e18a788a51_isr_945868_7.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/7f7bee1f-14c7-42f8-b5a9-37e18a788a51_isr_945868_7.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b313be17ab47027c752ec5466344831c with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash 9c6747afc7a9b88db64bb1dabb3afa1c parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/e3b678e3-d9fe-45c8-9660-6cdb8753dec1_characterizeImage_945755_57.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/e3b678e3-d9fe-45c8-9660-6cdb8753dec1_characterizeImage_945755_57.stderr parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/2efdbcfe-617a-4f43-a54e-fd049d581042_isr_945755_57.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/2efdbcfe-617a-4f43-a54e-fd049d581042_isr_945755_57.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 013f70142a6257ba30a9728416433f5a with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 41172e8940b9f2169e1cddd299a835c7 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/cd3ef40f-4bc5-48dc-bd2e-477b2a32df19_characterizeImage_945757_15.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/cd3ef40f-4bc5-48dc-bd2e-477b2a32df19_characterizeImage_945757_15.stderr parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/fc348cb2-b398-4eaf-9588-35a4febf349e_isr_945757_15.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/fc348cb2-b398-4eaf-9588-35a4febf349e_isr_945757_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:08 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e46d04753473156441e7f6f3267e6483 with result from task 1104 parsl.dataflow.dflow DEBUG: Task 1104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash 8ac2395c0d24c0218e7c7cffd6f68bb2 parsl.dataflow.memoization INFO: Task 1105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460be2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1105 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e1710e07-e59a-4ae1-9e8a-76422176ee15_characterizeImage_945867_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e1710e07-e59a-4ae1-9e8a-76422176ee15_characterizeImage_945867_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/263a7e79-0ca8-4aa5-996f-0036ead6faa7_isr_945867_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/263a7e79-0ca8-4aa5-996f-0036ead6faa7_isr_945867_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:09 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 396e6a5f83542d6f54b8e4a38415ded1 with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 7519a1f4246e4b66777f23449a7d297c parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/36eef5ea-60d7-4608-9ad4-2a96254bbbfc_characterizeImage_945757_48.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/36eef5ea-60d7-4608-9ad4-2a96254bbbfc_characterizeImage_945757_48.stderr parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/df8d1bcf-494f-4d16-871b-0f9f86619948_isr_945757_48.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/df8d1bcf-494f-4d16-871b-0f9f86619948_isr_945757_48.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9cc6305db387a23b28e5db0a127b1c1 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 914cb9f1d086afb9ca0909116fc346d3 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246392de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/07c2f948-bec9-4249-a2b4-1a087e4540d8_characterizeImage_945755_3.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/07c2f948-bec9-4249-a2b4-1a087e4540d8_characterizeImage_945755_3.stderr parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/812993e5-60a3-46e1-9da8-dca34e21d2fb_isr_945755_3.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/812993e5-60a3-46e1-9da8-dca34e21d2fb_isr_945755_3.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1fceee7aee9c8a321949774f873873e with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash de07a674c02bf436515a27e70b1980ff parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b6ad3ef9-5561-4030-addb-8ebb2e901e8e_characterizeImage_945756_29.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b6ad3ef9-5561-4030-addb-8ebb2e901e8e_characterizeImage_945756_29.stderr parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/d01f04e9-fad9-43ca-96bd-a3e86846dc11_isr_945756_29.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/d01f04e9-fad9-43ca-96bd-a3e86846dc11_isr_945756_29.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f63ffac279b90bc5c4b55c468e052182 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 08e0326b6388cd3bc6e195eb31b18119 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/8d904e3f-87f0-4602-b5f5-7c9327a328bf_characterizeImage_945757_61.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/8d904e3f-87f0-4602-b5f5-7c9327a328bf_characterizeImage_945757_61.stderr parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d3f0963b-4b13-4185-9ec5-1b916b61463c_isr_945757_61.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d3f0963b-4b13-4185-9ec5-1b916b61463c_isr_945757_61.stderr parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c80f40e617716d14c10708ed7f0aa166 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 95606d09081dbdeb4976eb73fe7bd9d7 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b5f91a35-5cc9-4507-aee2-30ae10146e58_characterizeImage_945756_12.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b5f91a35-5cc9-4507-aee2-30ae10146e58_characterizeImage_945756_12.stderr parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/ce353005-137b-484d-b822-0d740f982a28_isr_945756_12.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/ce353005-137b-484d-b822-0d740f982a28_isr_945756_12.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0af449bcb07a3f7dd64d2259255e98d1 with result from task 360 parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:09 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash fb366492bc0585715a2fa73bea9f2f7f parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dc81e187-8481-4f76-8629-056506af64b2_characterizeImage_945757_42.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dc81e187-8481-4f76-8629-056506af64b2_characterizeImage_945757_42.stderr parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/7ba95dfe-5790-4aed-a6aa-61e6c4622ac8_isr_945757_42.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/7ba95dfe-5790-4aed-a6aa-61e6c4622ac8_isr_945757_42.stderr parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 893f5aa3b50ad0ca100079ebe368f97b with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash dbef4b5b52e6eda873bbb75a1cacc1f7 parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6202145e-5725-4b2a-8be9-3aef559b5886_characterizeImage_945756_50.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6202145e-5725-4b2a-8be9-3aef559b5886_characterizeImage_945756_50.stderr parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0c571b4c-76ac-4b14-a7ff-2c05c03f99e8_isr_945756_50.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0c571b4c-76ac-4b14-a7ff-2c05c03f99e8_isr_945756_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:10 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:10 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 255cc49da3ec553fb7344b07461cf182 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash be28ee4d063cf481ed09994a64552bea parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dd0270c2-5486-4a9a-949b-c42465a79800_characterizeImage_945757_43.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dd0270c2-5486-4a9a-949b-c42465a79800_characterizeImage_945757_43.stderr parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/187dfb6a-7589-40c4-9e50-b34c7b0f6404_isr_945757_43.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/187dfb6a-7589-40c4-9e50-b34c7b0f6404_isr_945757_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:11 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:11 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:12 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f52e430944385c09c32ab4db6e822752 with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash de12a389fb6808c1e368e6f5540c5bb8 parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ccb92111-74ea-43f4-8251-0bfcddb30277_characterizeImage_945755_53.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ccb92111-74ea-43f4-8251-0bfcddb30277_characterizeImage_945755_53.stderr parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/45856bf5-c4e5-4db6-ab92-444bee26df11_isr_945755_53.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/45856bf5-c4e5-4db6-ab92-444bee26df11_isr_945755_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:12 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:13 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:13 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:14 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:14 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:15 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e97cfe3802b5fa19dd6760a90c021657 with result from task 879 parsl.dataflow.dflow DEBUG: Task 879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 880 has memoization hash 9c69737ae32e4ee942c169eb9d4b1e69 parsl.dataflow.memoization INFO: Task 880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461ab920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/626a2605-7b8b-4545-a657-b7a174cf891b_characterizeImage_945869_39.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/626a2605-7b8b-4545-a657-b7a174cf891b_characterizeImage_945869_39.stderr parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/c90c92c4-aa30-4e94-80ff-b8a1e07de967_isr_945869_39.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/c90c92c4-aa30-4e94-80ff-b8a1e07de967_isr_945869_39.stderr parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2afa95bb6c310e6d54fa374f6f584096 with result from task 858 parsl.dataflow.dflow DEBUG: Task 858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 100f0b7c17cac6f2773b09747ff83dd0 parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aa340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/6c0652fb-becd-4ca2-bd9c-59d4fca4e19c_characterizeImage_945755_40.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/6c0652fb-becd-4ca2-bd9c-59d4fca4e19c_characterizeImage_945755_40.stderr parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/dcf88211-7fa1-4748-b94b-58be40d1c3a2_isr_945755_40.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/dcf88211-7fa1-4748-b94b-58be40d1c3a2_isr_945755_40.stderr parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6984164a97c4ff5b78504547db12a5d8 with result from task 876 parsl.dataflow.dflow DEBUG: Task 876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 877 has memoization hash cdacfd17de94b32905e6d76d2fb4c248 parsl.dataflow.memoization INFO: Task 877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461ab600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/6c779dfe-4c78-461e-a826-3ecfa67f8984_characterizeImage_945755_28.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/6c779dfe-4c78-461e-a826-3ecfa67f8984_characterizeImage_945755_28.stderr parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/805bf1e4-d876-4ca6-97b6-0013b41490e1_isr_945755_28.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/805bf1e4-d876-4ca6-97b6-0013b41490e1_isr_945755_28.stderr parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddf6d3d9f8f3ef4e80619faeca5ac4d7 with result from task 870 parsl.dataflow.dflow DEBUG: Task 870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 871 has memoization hash 603b3ffe86fd527c1f763b708d6f81a2 parsl.dataflow.memoization INFO: Task 871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/1e5d8cd9-7ab9-485b-aa50-04393093bea5_characterizeImage_945867_26.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/1e5d8cd9-7ab9-485b-aa50-04393093bea5_characterizeImage_945867_26.stderr parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/0f5b7820-32dc-422e-b2a8-25e70919ecb6_isr_945867_26.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/0f5b7820-32dc-422e-b2a8-25e70919ecb6_isr_945867_26.stderr parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e7ef9505aedde1965082b79f2482a4f with result from task 861 parsl.dataflow.dflow DEBUG: Task 861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 862 has memoization hash 15f8bfc18aa2f1953fe63b61453fd46f parsl.dataflow.memoization INFO: Task 862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aa660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/a83343b1-6d18-453a-9832-2d877e00819b_characterizeImage_945867_28.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/a83343b1-6d18-453a-9832-2d877e00819b_characterizeImage_945867_28.stderr parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/0232bbc6-9a72-40e0-b10a-53baa0e2271d_isr_945867_28.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/0232bbc6-9a72-40e0-b10a-53baa0e2271d_isr_945867_28.stderr parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6f97255f52439a0ebb77d7989a0def5 with result from task 873 parsl.dataflow.dflow DEBUG: Task 873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash d3d5662fcf42535723b4aac3f04e0821 parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461ab2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/bd61a087-739f-42af-ba27-860ab0571b7d_characterizeImage_945755_8.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/bd61a087-739f-42af-ba27-860ab0571b7d_characterizeImage_945755_8.stderr parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b41c7e17-694d-4cf4-9a67-de5e71f8b661_isr_945755_8.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b41c7e17-694d-4cf4-9a67-de5e71f8b661_isr_945755_8.stderr parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd8ab94a1a9356aa5d2b8d6ce792871f with result from task 867 parsl.dataflow.dflow DEBUG: Task 867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 868 has memoization hash 092ca735b775d91093526df3b32b2ed1 parsl.dataflow.memoization INFO: Task 868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/cb88571a-3a1e-4ffa-b27b-53b1a0f0978e_characterizeImage_945869_19.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/cb88571a-3a1e-4ffa-b27b-53b1a0f0978e_characterizeImage_945869_19.stderr parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/05b541c2-e350-42e4-9cb7-a3237df7e701_isr_945869_19.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/05b541c2-e350-42e4-9cb7-a3237df7e701_isr_945869_19.stderr parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e0786bda7bad2fdc3cc56df2ec81d1a with result from task 864 parsl.dataflow.dflow DEBUG: Task 864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 865 has memoization hash 467131ae50d343ed807eab0cec46d99e parsl.dataflow.memoization INFO: Task 865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aa980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/fbdc54e8-c221-4b77-a743-ad6749aa42d8_characterizeImage_945868_54.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/fbdc54e8-c221-4b77-a743-ad6749aa42d8_characterizeImage_945868_54.stderr parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ffaf8af9-637c-4680-a244-c32aa53ef9e2_isr_945868_54.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ffaf8af9-637c-4680-a244-c32aa53ef9e2_isr_945868_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:16 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:16 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:35:17 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:35:17 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:17 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:18 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdbac11bab41c24a574f91148cb93a6d with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 0d79fac471875bf7986e3c531fa5ff1e parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246285a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/748ebc48-b4ce-470a-a052-078555c3b7f8_characterizeImage_945755_29.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/748ebc48-b4ce-470a-a052-078555c3b7f8_characterizeImage_945755_29.stderr parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3c4bda6c-36e9-426d-8b24-528e0fe608af_isr_945755_29.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3c4bda6c-36e9-426d-8b24-528e0fe608af_isr_945755_29.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8599cbab2ecc25ab73c6d4824461e0e5 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash c6262754a6cf900527e1cc26b2215027 parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/72164dd7-a3c8-40f8-b585-9748cfa8434b_characterizeImage_945755_21.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/72164dd7-a3c8-40f8-b585-9748cfa8434b_characterizeImage_945755_21.stderr parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/9f47f12d-d9b0-4199-8384-93a859efdf5b_isr_945755_21.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/9f47f12d-d9b0-4199-8384-93a859efdf5b_isr_945755_21.stderr parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dd268a60c3fa73e768b3c6effe7ae83 with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash cb58bfa58c77095466fcd662d95a8c14 parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462856c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/8da39226-0174-4b75-ad90-ac25fc7f185d_characterizeImage_945869_11.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/8da39226-0174-4b75-ad90-ac25fc7f185d_characterizeImage_945869_11.stderr parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/18f9268d-86ba-46f2-b7f6-40399a7af489_isr_945869_11.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/18f9268d-86ba-46f2-b7f6-40399a7af489_isr_945869_11.stderr parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2233b11b0be035c707556c79628518b with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 7ca68e4ec037fdbdc0d27b14dde82c24 parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246285da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/30acee16-a9db-4ab7-96a4-6b95d2934a2c_characterizeImage_945755_22.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/30acee16-a9db-4ab7-96a4-6b95d2934a2c_characterizeImage_945755_22.stderr parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3c75de45-b7b5-4d23-810b-45793d590b03_isr_945755_22.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3c75de45-b7b5-4d23-810b-45793d590b03_isr_945755_22.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbd3fbcaad4008d62d509da93eb79bd4 with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 8cd2fcad8352d0fcce6e25fb477457ba parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492464020c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/c11d1a60-564e-4871-9281-667f61b6921b_characterizeImage_945757_3.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/c11d1a60-564e-4871-9281-667f61b6921b_characterizeImage_945757_3.stderr parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/76d57c1d-8272-43a2-bf85-0e0ea65c06e8_isr_945757_3.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/76d57c1d-8272-43a2-bf85-0e0ea65c06e8_isr_945757_3.stderr parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3264e6bcb6c2aa1d53cc0d0a556cf664 with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 06083405b98d22de019a3c2470e3379c parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/5bc8eb4e-55c8-4502-b1e8-192cc0b1cd64_characterizeImage_945756_2.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/5bc8eb4e-55c8-4502-b1e8-192cc0b1cd64_characterizeImage_945756_2.stderr parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/54088db6-7a7a-4eb9-af38-f307cd031650_isr_945756_2.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/54088db6-7a7a-4eb9-af38-f307cd031650_isr_945756_2.stderr parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 581cbbfd7f7072046a84f666aba8c00a with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash b0f73d203e6d7ac3b83e82643bc11f36 parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/aaab5ec8-8be2-4c83-b097-1c5e407ee40c_characterizeImage_945756_57.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/aaab5ec8-8be2-4c83-b097-1c5e407ee40c_characterizeImage_945756_57.stderr parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/4cae27c9-96a4-4939-b84a-3233e38f6265_isr_945756_57.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/4cae27c9-96a4-4939-b84a-3233e38f6265_isr_945756_57.stderr parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fa296fdad75ecccd5ba53616226feab with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 756cbb1b4e6e8c59508fe0ed452e96e4 parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/68aa802c-962f-49f4-a85f-c8deff19a730_characterizeImage_945869_34.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/68aa802c-962f-49f4-a85f-c8deff19a730_characterizeImage_945869_34.stderr parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/0ab6ede6-03e9-4788-9689-5df7f4737801_isr_945869_34.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/0ab6ede6-03e9-4788-9689-5df7f4737801_isr_945869_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:18 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:19 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:19 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 810b68be655db961829705151bef731c with result from task 795 parsl.dataflow.dflow DEBUG: Task 795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash 0d786d0195d23e2063d063b900743bc5 parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246346160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/321beb0f-6910-4d56-b90a-a5753f96069d_characterizeImage_945869_7.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/321beb0f-6910-4d56-b90a-a5753f96069d_characterizeImage_945869_7.stderr parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/4daea850-2bc5-4ffe-b3fa-f3db6a07ed83_isr_945869_7.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/4daea850-2bc5-4ffe-b3fa-f3db6a07ed83_isr_945869_7.stderr parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bd86a68c9ae97ac555490e695ba78e6 with result from task 807 parsl.dataflow.dflow DEBUG: Task 807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 808 has memoization hash 8ce7668ac4576d5fa4e84a073c62dda4 parsl.dataflow.memoization INFO: Task 808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246346de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0ec401d0-05bf-41e8-9138-ec2d16aaad3f_characterizeImage_945868_57.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0ec401d0-05bf-41e8-9138-ec2d16aaad3f_characterizeImage_945868_57.stderr parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/35f66b15-a6f2-4ca1-a39e-f26609a99087_isr_945868_57.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/35f66b15-a6f2-4ca1-a39e-f26609a99087_isr_945868_57.stderr parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 196e0a16b2480100069c01d26de55e9f with result from task 771 parsl.dataflow.dflow DEBUG: Task 771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash e5543a28fb02c357605d9a3dbff59ab8 parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246313e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/3af11c3f-5b28-4ba4-898f-27f77d8e70af_characterizeImage_945868_22.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/3af11c3f-5b28-4ba4-898f-27f77d8e70af_characterizeImage_945868_22.stderr parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ef4dd465-f58b-4ddb-be4b-96ca43364e3b_isr_945868_22.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ef4dd465-f58b-4ddb-be4b-96ca43364e3b_isr_945868_22.stderr parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f39b10cfb0765bad502e4185784baf1 with result from task 786 parsl.dataflow.dflow DEBUG: Task 786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 787 has memoization hash 005f6d5f0e9227c8aad046fc51d79bca parsl.dataflow.memoization INFO: Task 787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246345800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/b3bb46ea-d941-4c5e-9770-dde2a8f71612_characterizeImage_945869_44.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/b3bb46ea-d941-4c5e-9770-dde2a8f71612_characterizeImage_945869_44.stderr parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/63c1a3e7-7bf1-41ff-b630-be601a4b2554_isr_945869_44.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/63c1a3e7-7bf1-41ff-b630-be601a4b2554_isr_945869_44.stderr parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87c28e0e68cbdad4de9758fcda54b841 with result from task 774 parsl.dataflow.dflow DEBUG: Task 774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 775 has memoization hash b6144d3f6199cf5e9df9828953672240 parsl.dataflow.memoization INFO: Task 775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f9cbb8a9-2503-467f-b14f-5742d8c0c764_characterizeImage_945756_21.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f9cbb8a9-2503-467f-b14f-5742d8c0c764_characterizeImage_945756_21.stderr parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/204a8a28-8805-4296-ab51-16910c3c489c_isr_945756_21.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/204a8a28-8805-4296-ab51-16910c3c489c_isr_945756_21.stderr parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a84b6bead43c8ff189ab99561bd4f8a2 with result from task 777 parsl.dataflow.dflow DEBUG: Task 777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 778 has memoization hash 12094b391d6ce2d1293d6e016c45d90f parsl.dataflow.memoization INFO: Task 778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/b8d5b54d-db75-4808-9cb4-479a2f80056e_characterizeImage_945755_43.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/b8d5b54d-db75-4808-9cb4-479a2f80056e_characterizeImage_945755_43.stderr parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e0c6f304-709c-4ea4-93e2-aa099718b803_isr_945755_43.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e0c6f304-709c-4ea4-93e2-aa099718b803_isr_945755_43.stderr parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9d7c8356f234e83dad7299643efad18 with result from task 783 parsl.dataflow.dflow DEBUG: Task 783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 784 has memoization hash 9346dc30e0cb5575844454c8b9015028 parsl.dataflow.memoization INFO: Task 784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463454e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/baff60b0-709f-4080-b56b-e02a6578393f_characterizeImage_945869_28.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/baff60b0-709f-4080-b56b-e02a6578393f_characterizeImage_945869_28.stderr parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/50d3b6a0-e9d8-43e0-adb3-9eb2df537649_isr_945869_28.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/50d3b6a0-e9d8-43e0-adb3-9eb2df537649_isr_945869_28.stderr parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ebe73dfe374f83810e8e8972045c719 with result from task 780 parsl.dataflow.dflow DEBUG: Task 780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 781 has memoization hash 78a31f13174ab687c87a5d2eab529252 parsl.dataflow.memoization INFO: Task 781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2fef4c39-3e7a-4a22-a8eb-62992a8c3a73_characterizeImage_945867_44.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2fef4c39-3e7a-4a22-a8eb-62992a8c3a73_characterizeImage_945867_44.stderr parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/a4a82628-37e8-4109-b9d7-c0bb33af44e9_isr_945867_44.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/a4a82628-37e8-4109-b9d7-c0bb33af44e9_isr_945867_44.stderr parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51da62051c1c633368590ed86130b695 with result from task 768 parsl.dataflow.dflow DEBUG: Task 768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 769 has memoization hash b9d61b774f15de3d111ccd294c7f1d52 parsl.dataflow.memoization INFO: Task 769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246313d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/240e9e59-915c-4f6e-bef2-1605cd20f2ba_characterizeImage_945756_42.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/240e9e59-915c-4f6e-bef2-1605cd20f2ba_characterizeImage_945756_42.stderr parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/4516ff7a-d29f-425e-b05e-978e21604a4b_isr_945756_42.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/4516ff7a-d29f-425e-b05e-978e21604a4b_isr_945756_42.stderr parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39cd4782b2f1c6a88f9dc3177298e723 with result from task 789 parsl.dataflow.dflow DEBUG: Task 789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 790 has memoization hash 764ed665ab2a8c39e3bb04cf980e07d3 parsl.dataflow.memoization INFO: Task 790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246345b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/a0cf1cef-8519-48f8-816d-08c54de7e0a7_characterizeImage_945756_1.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/a0cf1cef-8519-48f8-816d-08c54de7e0a7_characterizeImage_945756_1.stderr parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2db68b08-6f0c-4425-b417-a399a9527448_isr_945756_1.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2db68b08-6f0c-4425-b417-a399a9527448_isr_945756_1.stderr parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec66a6d96e89cec63f5511a3f5cf177c with result from task 798 parsl.dataflow.dflow DEBUG: Task 798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash 534d7ec3c0fe5b8439a362fffdc49488 parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246346480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/19e5fd0a-6567-45c2-89de-6b4872222f75_characterizeImage_945755_17.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/19e5fd0a-6567-45c2-89de-6b4872222f75_characterizeImage_945755_17.stderr parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/cb8e6113-79a1-475c-9c12-f33f2cd5ca77_isr_945755_17.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/cb8e6113-79a1-475c-9c12-f33f2cd5ca77_isr_945755_17.stderr parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 059eb0ff1b12c663d86011af31060e68 with result from task 801 parsl.dataflow.dflow DEBUG: Task 801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash e2b835e876e85314109e8f43680c2ee3 parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463467a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a95e41b9-93fb-4718-9dd3-370b390cf9bc_characterizeImage_945757_47.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a95e41b9-93fb-4718-9dd3-370b390cf9bc_characterizeImage_945757_47.stderr parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a53bb469-ac93-406f-973f-38750e6411ea_isr_945757_47.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a53bb469-ac93-406f-973f-38750e6411ea_isr_945757_47.stderr parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 964b7207f6848938a994735c798e85d1 with result from task 810 parsl.dataflow.dflow DEBUG: Task 810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 811 has memoization hash 21191dc2a63b000d300597eb803b9291 parsl.dataflow.memoization INFO: Task 811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/d2bd6043-267b-4e61-a296-80c2fbc509ba_characterizeImage_945868_14.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/d2bd6043-267b-4e61-a296-80c2fbc509ba_characterizeImage_945868_14.stderr parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/23598f03-eb09-4f22-bd65-191e7874b484_isr_945868_14.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/23598f03-eb09-4f22-bd65-191e7874b484_isr_945868_14.stderr parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 106a799cca7a1f000a845a5ab28820be with result from task 804 parsl.dataflow.dflow DEBUG: Task 804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash 4fb91040281baf89cdf2e276dc1b7b1a parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246346ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/fb159200-ba8d-426f-9652-dd160855f374_characterizeImage_945755_50.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/fb159200-ba8d-426f-9652-dd160855f374_characterizeImage_945755_50.stderr parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3023b4b3-ac97-4253-b91b-08145e4b1af5_isr_945755_50.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/3023b4b3-ac97-4253-b91b-08145e4b1af5_isr_945755_50.stderr parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 710e5c7d323e3d62134f4c1ddb1bc59e with result from task 813 parsl.dataflow.dflow DEBUG: Task 813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 814 has memoization hash a25b5127ec4846857474650b4bda02b3 parsl.dataflow.memoization INFO: Task 814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/7d3cf7ac-88a3-4575-8499-3f5a03b83a36_characterizeImage_945756_39.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/7d3cf7ac-88a3-4575-8499-3f5a03b83a36_characterizeImage_945756_39.stderr parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/c098b796-0054-4cb6-9648-60ac9d40c7c5_isr_945756_39.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/c098b796-0054-4cb6-9648-60ac9d40c7c5_isr_945756_39.stderr parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3a9ee2772b27306801bcfe92142820f with result from task 792 parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:20 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow DEBUG: Task 792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash 18e91ca72d1d1174b90b1f4c6861f95e parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246345e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/1b62231d-cf50-4958-915b-388d6f09dbe6_characterizeImage_945868_21.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/1b62231d-cf50-4958-915b-388d6f09dbe6_characterizeImage_945868_21.stderr parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d548272e-48ee-46d3-aa4c-4d59948b0eb4_isr_945868_21.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d548272e-48ee-46d3-aa4c-4d59948b0eb4_isr_945868_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:20 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:21 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:21 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:22 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:22 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 799b455dba89a17f21e8cea74092c280 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash bf8a8828c26257c2cb05f05624a7f25a parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d2b9f1f6-cc6e-4109-a3f1-991d49b09293_calibrate_945755_37.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d2b9f1f6-cc6e-4109-a3f1-991d49b09293_calibrate_945755_37.stderr parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/3db0162c-a36b-4b1e-9680-3a59e52f3c9d_characterizeImage_945755_37.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/3db0162c-a36b-4b1e-9680-3a59e52f3c9d_characterizeImage_945755_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:23 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:23 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:24 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:25 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a8208acd2e328d558d9e01f1ec5b1f5 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 3f70afbc4cc0e33b38e25698bc9590f8 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246391620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/21f0b060-9829-4aeb-9c89-bba7f45bc498_calibrate_945755_39.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/21f0b060-9829-4aeb-9c89-bba7f45bc498_calibrate_945755_39.stderr parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ed18a17c-6bcd-4107-a214-74c69cdd73e9_characterizeImage_945755_39.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ed18a17c-6bcd-4107-a214-74c69cdd73e9_characterizeImage_945755_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:25 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:26 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:26 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbb286924fefc21cd3a8635dce3ce3c3 with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash b2386eb0660300070584b8a0a01ab4dc parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246311300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/a839ef76-b482-453f-b06b-e1310a574441_calibrate_945755_35.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/a839ef76-b482-453f-b06b-e1310a574441_calibrate_945755_35.stderr parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ce85a6a2-2c82-4e6f-85c4-aee266e25f11_characterizeImage_945755_35.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ce85a6a2-2c82-4e6f-85c4-aee266e25f11_characterizeImage_945755_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:27 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:27 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:28 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 142 try 0 failed parsl.dataflow.dflow INFO: Task 142 marked for retry parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2f3b6492-8334-4710-8723-08ee991967ae_characterizeImage_945867_2.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2f3b6492-8334-4710-8723-08ee991967ae_characterizeImage_945867_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash ab4632a424ad3db76f806f61db48d96a parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 1 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2f3b6492-8334-4710-8723-08ee991967ae_characterizeImage_945867_2.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2f3b6492-8334-4710-8723-08ee991967ae_characterizeImage_945867_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:28 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 613 try 0 failed parsl.dataflow.dflow INFO: Task 613 marked for retry parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/5bc8eb4e-55c8-4502-b1e8-192cc0b1cd64_characterizeImage_945756_2.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/5bc8eb4e-55c8-4502-b1e8-192cc0b1cd64_characterizeImage_945756_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 06083405b98d22de019a3c2470e3379c parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 1 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/5bc8eb4e-55c8-4502-b1e8-192cc0b1cd64_characterizeImage_945756_2.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/5bc8eb4e-55c8-4502-b1e8-192cc0b1cd64_characterizeImage_945756_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:29 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:29 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:30 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:30 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:31 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:31 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8062151b6a1de6af2fa681f104a6d10f with result from task 954 parsl.dataflow.dflow DEBUG: Task 954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 955 has memoization hash e0ebff53a5c0a35ff550a8bc14442f03 parsl.dataflow.memoization INFO: Task 955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/0048233f-a093-4d88-ad51-ddd5defcfca0_characterizeImage_945755_54.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/0048233f-a093-4d88-ad51-ddd5defcfca0_characterizeImage_945755_54.stderr parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b843959c-4ed7-46ad-83e2-c3af90cd0147_isr_945755_54.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b843959c-4ed7-46ad-83e2-c3af90cd0147_isr_945755_54.stderr parsl.dataflow.dflow INFO: Task 957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9953bf5727e6cf095573a927b73b245 with result from task 957 parsl.dataflow.dflow DEBUG: Task 957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 958 has memoization hash 3692aafd50f95bcba72e2d0cda1b420d parsl.dataflow.memoization INFO: Task 958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/419e4477-d32e-492a-8420-11a76d73df8a_characterizeImage_945756_23.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/419e4477-d32e-492a-8420-11a76d73df8a_characterizeImage_945756_23.stderr parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/60f05eed-49ac-4181-b793-45ac9c6782bf_isr_945756_23.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/60f05eed-49ac-4181-b793-45ac9c6782bf_isr_945756_23.stderr parsl.dataflow.dflow INFO: Task 936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21051a311d2a06cd852eb87e15529f87 with result from task 936 parsl.dataflow.dflow DEBUG: Task 936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 937 has memoization hash fb097d84c298f786e0a7cb2483b2e3f3 parsl.dataflow.memoization INFO: Task 937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246228900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/310b0d1b-2f32-45a4-8e12-af194b96803d_characterizeImage_945869_43.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/310b0d1b-2f32-45a4-8e12-af194b96803d_characterizeImage_945869_43.stderr parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/ffec0a83-fe20-419c-a5ba-60defbe5b179_isr_945869_43.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/ffec0a83-fe20-419c-a5ba-60defbe5b179_isr_945869_43.stderr parsl.dataflow.dflow INFO: Task 951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa3afc141b77fd73a74cef09096f1c88 with result from task 951 parsl.dataflow.dflow DEBUG: Task 951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 952 has memoization hash 130ba1b07dce851ebdae533cf53989a1 parsl.dataflow.memoization INFO: Task 952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/2d1bcfe6-b009-42af-a161-4cbc4ccadd5e_characterizeImage_945755_18.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/2d1bcfe6-b009-42af-a161-4cbc4ccadd5e_characterizeImage_945755_18.stderr parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/48a69363-75c7-4e14-8206-41e02853461e_isr_945755_18.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/48a69363-75c7-4e14-8206-41e02853461e_isr_945755_18.stderr parsl.dataflow.dflow INFO: Task 948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90b84b644afde9061520413fee8a88a2 with result from task 948 parsl.dataflow.dflow DEBUG: Task 948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 949 has memoization hash f2b024db7559228c0f48489820ae2486 parsl.dataflow.memoization INFO: Task 949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/cf509cec-6cae-40b4-a7ae-86c110c3df1e_characterizeImage_945869_21.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/cf509cec-6cae-40b4-a7ae-86c110c3df1e_characterizeImage_945869_21.stderr parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/7bb88b26-7aeb-4843-ac42-5006a4fc8c88_isr_945869_21.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/7bb88b26-7aeb-4843-ac42-5006a4fc8c88_isr_945869_21.stderr parsl.dataflow.dflow INFO: Task 945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcc740ba6ea8611941d12a6f9652eb26 with result from task 945 parsl.dataflow.dflow DEBUG: Task 945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 946 has memoization hash 986ab2d98d480de5c27c3b7627d79696 parsl.dataflow.memoization INFO: Task 946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246229da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b67267ac-306c-4b76-aea9-f4130a9badb7_characterizeImage_945756_32.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b67267ac-306c-4b76-aea9-f4130a9badb7_characterizeImage_945756_32.stderr parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/3f8058e7-6956-41ca-9a20-63ebd617384e_isr_945756_32.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/3f8058e7-6956-41ca-9a20-63ebd617384e_isr_945756_32.stderr parsl.dataflow.dflow INFO: Task 942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 071a1d29bebdd92c419fb0d900f99a03 with result from task 942 parsl.dataflow.dflow DEBUG: Task 942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 943 has memoization hash 5541fad15dc5d169a3d3c34bceb5d1bd parsl.dataflow.memoization INFO: Task 943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246229a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/d2dcceb0-ef83-41ab-b355-340164ee6ac9_characterizeImage_945756_51.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/d2dcceb0-ef83-41ab-b355-340164ee6ac9_characterizeImage_945756_51.stderr parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/ac0f42b7-0c81-422c-9bd2-a5ebd91fa512_isr_945756_51.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/ac0f42b7-0c81-422c-9bd2-a5ebd91fa512_isr_945756_51.stderr parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ce16e88ef3a7709ae26f771b9ddb989 with result from task 747 parsl.dataflow.dflow DEBUG: Task 747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 748 has memoization hash 528caa1c4f6db5164d9a90433d67906f parsl.dataflow.memoization INFO: Task 748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246312700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e67a3eaf-de9e-4df2-9d18-64888ad2263c_characterizeImage_945867_53.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e67a3eaf-de9e-4df2-9d18-64888ad2263c_characterizeImage_945867_53.stderr parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/f33c0f00-bd9b-4060-94a7-93e857e7f9f2_isr_945867_53.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/f33c0f00-bd9b-4060-94a7-93e857e7f9f2_isr_945867_53.stderr parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f65420212237dbcaa93e9013e317045 with result from task 756 parsl.dataflow.dflow DEBUG: Task 756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 757 has memoization hash 7132407521d9775c491cdaebcdb0d5a7 parsl.dataflow.memoization INFO: Task 757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246312fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/a002c301-c0b3-4b5f-a43a-4875774ad2d7_characterizeImage_945867_47.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/a002c301-c0b3-4b5f-a43a-4875774ad2d7_characterizeImage_945867_47.stderr parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/940396b7-6cae-4d80-b309-0c2ddae0f919_isr_945867_47.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/940396b7-6cae-4d80-b309-0c2ddae0f919_isr_945867_47.stderr parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f18e3c2d750aae7efe0f9bb51707a2e7 with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash a245db905477be0ae805fd179691e850 parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246287ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/19bcabc0-ee98-439d-9cf7-df052131c5c8_characterizeImage_945756_14.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/19bcabc0-ee98-439d-9cf7-df052131c5c8_characterizeImage_945756_14.stderr parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/eba12411-e3ec-48cc-8a5f-be3a65ae55de_isr_945756_14.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/eba12411-e3ec-48cc-8a5f-be3a65ae55de_isr_945756_14.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c993509658107e92d1014fff56f0323a with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash 81ab2da9fb3a3495ae5212b6265bd602 parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246287ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/935de672-7f32-44c4-b943-17e3a5f92d8f_characterizeImage_945755_60.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/935de672-7f32-44c4-b943-17e3a5f92d8f_characterizeImage_945755_60.stderr parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8185400d-1be4-426b-a705-c4e14c785100_isr_945755_60.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8185400d-1be4-426b-a705-c4e14c785100_isr_945755_60.stderr parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef6d3ab84880e7d58722c8e8fd7ccbb2 with result from task 762 parsl.dataflow.dflow DEBUG: Task 762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash 0b886966824120c00b9e96b1c8652bd7 parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246313600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0c89c7fe-5d97-460b-bcc1-9df1c4a37131_characterizeImage_945868_2.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0c89c7fe-5d97-460b-bcc1-9df1c4a37131_characterizeImage_945868_2.stderr parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/606249f0-2a0d-4dfd-be4b-7ee79632e543_isr_945868_2.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/606249f0-2a0d-4dfd-be4b-7ee79632e543_isr_945868_2.stderr parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9621a5b5d123c40c73d24661356f55dc with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 9b76c30be793c7dc418bd5171cc3ada7 parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246287100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/8550d854-c3cb-4b25-90f7-da81a74bea6d_characterizeImage_945868_5.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/8550d854-c3cb-4b25-90f7-da81a74bea6d_characterizeImage_945868_5.stderr parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/7cd86ec1-e970-4225-b864-9d537c473b1b_isr_945868_5.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/7cd86ec1-e970-4225-b864-9d537c473b1b_isr_945868_5.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c92d50a6e9b5ea08ae75136b78c5136 with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash c7019a67d51883d54a92076151805609 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/df86df58-d380-460d-820f-dca9138852a1_characterizeImage_945867_1.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/df86df58-d380-460d-820f-dca9138852a1_characterizeImage_945867_1.stderr parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1cd2e6ae-4dfe-48fb-acc6-5f2a2a845483_isr_945867_1.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/1cd2e6ae-4dfe-48fb-acc6-5f2a2a845483_isr_945867_1.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1112d4de0f250afcb1849320fa4ccee1 with result from task 648 parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:32 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 0c42cc810eff1b59ba52f944951a0c52 parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246287f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/fe524c3b-a8e4-4124-bb30-deaf33fbb7ee_characterizeImage_945868_36.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/fe524c3b-a8e4-4124-bb30-deaf33fbb7ee_characterizeImage_945868_36.stderr parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/6ea841dc-e970-4156-a676-a780d923c1b0_isr_945868_36.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/6ea841dc-e970-4156-a676-a780d923c1b0_isr_945868_36.stderr parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26cef781469d527deb0d874b555264a2 with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 664bc35da0953bb55916ac0f66967e69 parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246287740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/31238fe6-3138-499b-aa4a-2a1c2811b2a4_characterizeImage_945867_49.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/31238fe6-3138-499b-aa4a-2a1c2811b2a4_characterizeImage_945867_49.stderr parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/6e8992ec-c889-4bb6-84f2-445ff625cea8_isr_945867_49.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/6e8992ec-c889-4bb6-84f2-445ff625cea8_isr_945867_49.stderr parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0162f74766389308b30a0e5eb90231e0 with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash c40b76db24a47127af5ac1c749c7e41d parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4d4b5088-6502-4984-9184-721152ed8fc6_characterizeImage_945756_22.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4d4b5088-6502-4984-9184-721152ed8fc6_characterizeImage_945756_22.stderr parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/c972aef3-1931-4d45-b088-073be10f39f5_isr_945756_22.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/c972aef3-1931-4d45-b088-073be10f39f5_isr_945756_22.stderr parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51580768b7049e1f199a67acadeb3016 with result from task 765 parsl.dataflow.dflow DEBUG: Task 765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 766 has memoization hash c40734718c2aedaf3d081a33e0be42f1 parsl.dataflow.memoization INFO: Task 766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246313a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/c400edc2-1a4d-49c6-8556-610471e659c6_characterizeImage_945755_25.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/c400edc2-1a4d-49c6-8556-610471e659c6_characterizeImage_945755_25.stderr parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b77c33c5-1683-4103-bf98-a34beec564ff_isr_945755_25.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b77c33c5-1683-4103-bf98-a34beec564ff_isr_945755_25.stderr parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1950c646a6a74313b71401cc8cfe3a0 with result from task 744 parsl.dataflow.dflow DEBUG: Task 744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash f2935b934b537d51201004b748ef9caf parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463120c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/a8cc5b78-0b87-4953-8ee0-c80df3064dcd_characterizeImage_945756_52.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/a8cc5b78-0b87-4953-8ee0-c80df3064dcd_characterizeImage_945756_52.stderr parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/69549d2a-acc6-4939-8b1a-0a31cf46434c_isr_945756_52.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/69549d2a-acc6-4939-8b1a-0a31cf46434c_isr_945756_52.stderr parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9581b3a55deb4d488a7e02a116042ed9 with result from task 759 parsl.dataflow.dflow DEBUG: Task 759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 760 has memoization hash d666fdcdbada8651f5570bf9e80e559d parsl.dataflow.memoization INFO: Task 760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463132e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/d6d4a07b-e781-4f94-b590-48b17e2dabcb_characterizeImage_945756_59.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/d6d4a07b-e781-4f94-b590-48b17e2dabcb_characterizeImage_945756_59.stderr parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/c7f1cef7-3ccc-489b-92a5-f9d91a4fd039_isr_945756_59.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/c7f1cef7-3ccc-489b-92a5-f9d91a4fd039_isr_945756_59.stderr parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20d822a3d79e122020d29bf6aaacf302 with result from task 750 parsl.dataflow.dflow DEBUG: Task 750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 751 has memoization hash cb06820246f47129852434df70c2fe94 parsl.dataflow.memoization INFO: Task 751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246312ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/c8de9a78-6da7-42c0-b7bf-500ad82c1870_characterizeImage_945868_18.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/c8de9a78-6da7-42c0-b7bf-500ad82c1870_characterizeImage_945868_18.stderr parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/cbc870ed-fd0d-44e1-9647-7820621835f9_isr_945868_18.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/cbc870ed-fd0d-44e1-9647-7820621835f9_isr_945868_18.stderr parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dceaa48692d5327a4236b8c5ca87333 with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash 02ddb1959c42d5d4a5e48dc22602a3a8 parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246287420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/f8e7cad5-34f7-4672-9963-bc716c015bd7_characterizeImage_945755_9.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/f8e7cad5-34f7-4672-9963-bc716c015bd7_characterizeImage_945755_9.stderr parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/13257298-71c7-4af5-a9dd-2b7b7335bcb7_isr_945755_9.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/13257298-71c7-4af5-a9dd-2b7b7335bcb7_isr_945755_9.stderr parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc4d65a324c14fcb6ee53702b2b87575 with result from task 753 parsl.dataflow.dflow DEBUG: Task 753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 754 has memoization hash 00620e11b673f181d001d9366cc2b98e parsl.dataflow.memoization INFO: Task 754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246312de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/e167eab4-fd24-403e-8180-30daf7ae4bb9_characterizeImage_945756_37.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/e167eab4-fd24-403e-8180-30daf7ae4bb9_characterizeImage_945756_37.stderr parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/169a4eb0-d14e-4ddd-9423-37d8d1dcf650_isr_945756_37.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/169a4eb0-d14e-4ddd-9423-37d8d1dcf650_isr_945756_37.stderr parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a69e8612de3734ce331e0d5d9ff86e2 with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash bda45173f8d6e9f7000f6aa24eac1768 parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463104a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/e4b3104e-fbc9-4eb2-8c1e-152e60e965f6_characterizeImage_945756_41.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/e4b3104e-fbc9-4eb2-8c1e-152e60e965f6_characterizeImage_945756_41.stderr parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2a0ed1c6-f1fc-4fa3-9ecf-62226db517b8_isr_945756_41.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2a0ed1c6-f1fc-4fa3-9ecf-62226db517b8_isr_945756_41.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 190dcdee71367cafe1f8c41dcf10f421 with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash c32d831971062837dd8b1703e99ed644 parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/4869a680-bf71-4cb6-b385-138d029eac3f_characterizeImage_945868_34.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/4869a680-bf71-4cb6-b385-138d029eac3f_characterizeImage_945868_34.stderr parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d6396833-efbe-49bb-b036-a6090ecd76c5_isr_945868_34.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/d6396833-efbe-49bb-b036-a6090ecd76c5_isr_945868_34.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4893f16b3dbcbaedf97231b4c15aa7fe with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash 8ba4ba88ee730a79b6d871ca04b60824 parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463111c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/9cc09445-898c-457f-9698-ab4daf10eedc_characterizeImage_945869_3.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/9cc09445-898c-457f-9698-ab4daf10eedc_characterizeImage_945869_3.stderr parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/434ca6db-14af-4874-8c1f-5ba790c0f95d_isr_945869_3.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/434ca6db-14af-4874-8c1f-5ba790c0f95d_isr_945869_3.stderr parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04cb5e9026cfa140b47e614d1ea37fb7 with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash 149de646a1f497701010fcf89285d57d parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246310ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/b58e0d25-249c-44fb-8734-7df777dab11d_characterizeImage_945755_26.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/b58e0d25-249c-44fb-8734-7df777dab11d_characterizeImage_945755_26.stderr parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d7c56008-c0d8-4be2-b735-ac62b6e16dc0_isr_945755_26.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d7c56008-c0d8-4be2-b735-ac62b6e16dc0_isr_945755_26.stderr parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16ea903d45bf9b9e7215238a8ada17a3 with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 02a4b2f007dcd34e7c48717ace6d0698 parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246310ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/767e3b0f-3ec9-487e-a2cd-46b019d38825_characterizeImage_945868_3.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/767e3b0f-3ec9-487e-a2cd-46b019d38825_characterizeImage_945868_3.stderr parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/28f74549-c3dd-49fa-90ae-e84ca71d1d2c_isr_945868_3.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/28f74549-c3dd-49fa-90ae-e84ca71d1d2c_isr_945868_3.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b75520d35045bd824781c6505a9b4293 with result from task 705 parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:32 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash e340444fac2be8aa3934005571e62633 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246285e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/3ee8d640-819e-484d-ba4d-52b2fc118b31_characterizeImage_945756_18.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/3ee8d640-819e-484d-ba4d-52b2fc118b31_characterizeImage_945756_18.stderr parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0d2c3011-2a86-4e49-9293-56033ab02a8e_isr_945756_18.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/0d2c3011-2a86-4e49-9293-56033ab02a8e_isr_945756_18.stderr parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d545e92ec678cbe9b7c416701cf7dc1c with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash 42fb0f081e23a6395ab9cc382709199f parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/3dd4ea47-fd9d-4e1c-b014-75061e41f46f_characterizeImage_945757_11.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/3dd4ea47-fd9d-4e1c-b014-75061e41f46f_characterizeImage_945757_11.stderr parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/32e75895-cf20-43f9-813b-d765bcc90502_isr_945757_11.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/32e75895-cf20-43f9-813b-d765bcc90502_isr_945757_11.stderr parsl.dataflow.dflow INFO: Task 939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c583c18c8e3f885cd8962b830db93ae with result from task 939 parsl.dataflow.dflow DEBUG: Task 939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 940 has memoization hash 6d13ac42f0cda595186584a92e4958b2 parsl.dataflow.memoization INFO: Task 940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462285e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/38a89c5d-a79c-4dd8-8666-163d051adf76_characterizeImage_945757_58.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/38a89c5d-a79c-4dd8-8666-163d051adf76_characterizeImage_945757_58.stderr parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d5684a72-352d-406d-aa70-1f42cdeedaff_isr_945757_58.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d5684a72-352d-406d-aa70-1f42cdeedaff_isr_945757_58.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 755679b5de4a76e02deedc2e3a365518 with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 6aa3a474c1c9528a2d9aad65af350949 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d2943037-aecd-44e2-b128-0d53f86ea310_calibrate_945757_39.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d2943037-aecd-44e2-b128-0d53f86ea310_calibrate_945757_39.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/40373139-de3b-462c-8da4-779177b77ec5_characterizeImage_945757_39.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/40373139-de3b-462c-8da4-779177b77ec5_characterizeImage_945757_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:33 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:33 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e6a5efd6dd6bf31edae0f52e2a997d3 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash b35df9d9cfe607960c9af6c3675e3641 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479944a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/ac0b9dcc-becb-4520-bdbf-b6db41dc809b_calibrate_945756_36.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/ac0b9dcc-becb-4520-bdbf-b6db41dc809b_calibrate_945756_36.stderr parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6feaebef-61d1-4fa6-a86c-a02d1586bde4_characterizeImage_945756_36.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6feaebef-61d1-4fa6-a86c-a02d1586bde4_characterizeImage_945756_36.stderr parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69ac604a2186e3c014f2d764aa1b2d98 with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash 776f21bc713c93a51f6f8998dd65ab12 parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/964b792c-4e1f-4683-8c7f-c280b79dce14_calibrate_945756_46.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/964b792c-4e1f-4683-8c7f-c280b79dce14_calibrate_945756_46.stderr parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/453eaf01-4ca6-4f5a-95df-6a311f5d3a18_characterizeImage_945756_46.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/453eaf01-4ca6-4f5a-95df-6a311f5d3a18_characterizeImage_945756_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:34 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:34 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:34 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3108a0baeb77a7c2424d337d0fa41961 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 323623307ba2d258adaa8e8c5fc5d369 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/360b4176-df6e-40d0-a820-760ad7ecdc86_calibrate_945755_55.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/360b4176-df6e-40d0-a820-760ad7ecdc86_calibrate_945755_55.stderr parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/2a34b4b9-cc64-4dd7-8f1c-42a353ad390f_characterizeImage_945755_55.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/2a34b4b9-cc64-4dd7-8f1c-42a353ad390f_characterizeImage_945755_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:35 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f84c078aab1fef2fc64fe42e1e25891 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 57a8ec3b0a1622eafd0aacbcec04c5b6 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/8cf38951-bcd9-4b5b-b27b-cf0a581b64f1_calibrate_945757_51.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/8cf38951-bcd9-4b5b-b27b-cf0a581b64f1_calibrate_945757_51.stderr parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b84f6c29-0de1-49ec-a0dd-967112729d96_characterizeImage_945757_51.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b84f6c29-0de1-49ec-a0dd-967112729d96_characterizeImage_945757_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:35 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:36 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:37 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9dca989126e1e42255b3cf7028952d1 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 47b2afdda48fe787cbccc1abba4c2f6a parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/d99990e1-9ac7-4706-adc3-331bbb8140e0_calibrate_945756_26.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/d99990e1-9ac7-4706-adc3-331bbb8140e0_calibrate_945756_26.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/a3883c77-df8d-4a56-a1f5-c319121e2bae_characterizeImage_945756_26.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/a3883c77-df8d-4a56-a1f5-c319121e2bae_characterizeImage_945756_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:37 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb9673fc39ee61d7deab98dc805a5401 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 7c5b2a26ea3ee2a5d4b3e1f628825369 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f701bd8d-d39f-4f4e-b771-3cde81de548a_calibrate_945755_52.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f701bd8d-d39f-4f4e-b771-3cde81de548a_calibrate_945755_52.stderr parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ca01fbce-00bb-4888-9056-798b54ecc109_characterizeImage_945755_52.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ca01fbce-00bb-4888-9056-798b54ecc109_characterizeImage_945755_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:38 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:38 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:38 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a236adcc87c8a116c586e314b6cc154d with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 97497bf69b536c480942b9cf234930d5 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/1fc09c80-ae30-48e1-af7a-c113f0ffc03b_calibrate_945757_22.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/1fc09c80-ae30-48e1-af7a-c113f0ffc03b_calibrate_945757_22.stderr parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/1b1172e6-671c-44ce-be16-a76425aeb6b7_characterizeImage_945757_22.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/1b1172e6-671c-44ce-be16-a76425aeb6b7_characterizeImage_945757_22.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f2552bfc15a8e31bd313c626c512a5a with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 4cabb54369a8aa41d001e4fcda373413 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/bf115159-9fd7-40a0-8783-e2d1be586bfd_calibrate_945757_34.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/bf115159-9fd7-40a0-8783-e2d1be586bfd_calibrate_945757_34.stderr parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/9c262849-fd75-4efc-88be-8b46d9183802_characterizeImage_945757_34.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/9c262849-fd75-4efc-88be-8b46d9183802_characterizeImage_945757_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:38 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:39 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:39 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:40 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 1071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e2d83c5794fd367ef8e716c4e9317a0 with result from task 1071 parsl.dataflow.dflow DEBUG: Task 1071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash 3b9a3a049d7e24497a2afb529882ac28 parsl.dataflow.memoization INFO: Task 1072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1072 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/0ff7d598-bbcc-4555-8044-8112c607c7f5_characterizeImage_945867_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/0ff7d598-bbcc-4555-8044-8112c607c7f5_characterizeImage_945867_4.stderr parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/aeb10ca6-6258-467f-8ab5-92d74cb3cb88_isr_945867_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/aeb10ca6-6258-467f-8ab5-92d74cb3cb88_isr_945867_4.stderr parsl.dataflow.dflow INFO: Task 1077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f8d6f4c47f37cdaa99d21b1eaad090b with result from task 1077 parsl.dataflow.dflow DEBUG: Task 1077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1078 has memoization hash f919f07e159d973a2a477fe7f444daf7 parsl.dataflow.memoization INFO: Task 1078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1078 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/d2038f1a-053c-4fdd-a3bf-ae24f4d667ca_characterizeImage_945867_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/d2038f1a-053c-4fdd-a3bf-ae24f4d667ca_characterizeImage_945867_51.stderr parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/6ccf2c1a-bd61-42e4-a61f-df86db50f873_isr_945867_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/6ccf2c1a-bd61-42e4-a61f-df86db50f873_isr_945867_51.stderr parsl.dataflow.dflow INFO: Task 1062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e250870bfaf6c4d9aa81572ad828b3cb with result from task 1062 parsl.dataflow.dflow DEBUG: Task 1062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash f9917cb3d5a2d05e5eca33b80fb9cd4a parsl.dataflow.memoization INFO: Task 1063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1063 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/3eae7396-a860-43be-9c75-20db509ecc36_characterizeImage_945869_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/3eae7396-a860-43be-9c75-20db509ecc36_characterizeImage_945869_42.stderr parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/5cf332fb-e191-4cf0-88a0-198598412807_isr_945869_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/5cf332fb-e191-4cf0-88a0-198598412807_isr_945869_42.stderr parsl.dataflow.dflow INFO: Task 1050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eba1b9633ce358b479aee923845901d8 with result from task 1050 parsl.dataflow.dflow DEBUG: Task 1050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash a37cb82effb4b3ee4b0e4ce22353a45f parsl.dataflow.memoization INFO: Task 1051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/45e915ff-829a-403e-b624-b82f2159ff0b_characterizeImage_945868_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/45e915ff-829a-403e-b624-b82f2159ff0b_characterizeImage_945868_55.stderr parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ed802df7-7549-4a4b-9d80-6b353b5accfb_isr_945868_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/ed802df7-7549-4a4b-9d80-6b353b5accfb_isr_945868_55.stderr parsl.dataflow.dflow INFO: Task 1047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23e1cf3abed6a853d22bb1b8fad98837 with result from task 1047 parsl.dataflow.dflow DEBUG: Task 1047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash b1ba77786057c5b5350fd5df5491bcc6 parsl.dataflow.memoization INFO: Task 1048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4121fd82-9e81-4930-b739-e9c03ec4c8d8_characterizeImage_945756_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4121fd82-9e81-4930-b739-e9c03ec4c8d8_characterizeImage_945756_4.stderr parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/e3b3e3b4-97ef-41e8-aaa9-137beae41810_isr_945756_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/e3b3e3b4-97ef-41e8-aaa9-137beae41810_isr_945756_4.stderr parsl.dataflow.dflow INFO: Task 1068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5608bde9512c301dfe0e5a200ce852d7 with result from task 1068 parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:40 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow DEBUG: Task 1068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1069 has memoization hash d2496cd3e726eba274090c149d582d4e parsl.dataflow.memoization INFO: Task 1069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1069 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/043316c7-0d58-412d-b3f9-aa511f7c47aa_characterizeImage_945868_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/043316c7-0d58-412d-b3f9-aa511f7c47aa_characterizeImage_945868_59.stderr parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/97e84b8e-cbea-4728-9263-d46a66e7163b_isr_945868_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/97e84b8e-cbea-4728-9263-d46a66e7163b_isr_945868_59.stderr parsl.dataflow.dflow INFO: Task 1074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bef5395e5111c9bacc0d56a2bf8544f with result from task 1074 parsl.dataflow.dflow DEBUG: Task 1074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1075 has memoization hash fdd4b85da71d3ddfaf32b5368639ea8d parsl.dataflow.memoization INFO: Task 1075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1075 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/0e65992e-3e38-41ce-80ad-11c9e6324f4e_characterizeImage_945869_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/0e65992e-3e38-41ce-80ad-11c9e6324f4e_characterizeImage_945869_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8bdc49a1-783b-49cc-8a94-d880bb890129_isr_945869_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/8bdc49a1-783b-49cc-8a94-d880bb890129_isr_945869_61.stderr parsl.dataflow.dflow INFO: Task 1092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63cb4a923f207bb0020506ebdbcd037a with result from task 1092 parsl.dataflow.dflow DEBUG: Task 1092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1093 has memoization hash 5451cd1a46485afba39607e3fb84a742 parsl.dataflow.memoization INFO: Task 1093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1093 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/2404b549-8be8-479e-8a78-e40964bc5e6c_characterizeImage_945757_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/2404b549-8be8-479e-8a78-e40964bc5e6c_characterizeImage_945757_4.stderr parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/6775f9d3-08f0-4a94-b523-d66058378b00_isr_945757_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/6775f9d3-08f0-4a94-b523-d66058378b00_isr_945757_4.stderr parsl.dataflow.dflow INFO: Task 1089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5091682e979abdf310e486a56af7902c with result from task 1089 parsl.dataflow.dflow DEBUG: Task 1089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1090 has memoization hash eae6a8a7537f975606af42340f0af47a parsl.dataflow.memoization INFO: Task 1090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1090 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/8e027e21-2443-4d11-840d-4867ec72841a_characterizeImage_945755_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/8e027e21-2443-4d11-840d-4867ec72841a_characterizeImage_945755_31.stderr parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e63bef9f-1bed-4de7-98f5-7f311ce81047_isr_945755_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e63bef9f-1bed-4de7-98f5-7f311ce81047_isr_945755_31.stderr parsl.dataflow.dflow INFO: Task 1086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f44f3d1d353075490aaacdcfc2827e4 with result from task 1086 parsl.dataflow.dflow DEBUG: Task 1086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1087 has memoization hash c8e03885fb5d100ba73952b593196684 parsl.dataflow.memoization INFO: Task 1087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1087 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/1ca0a5c6-34b7-44b7-ae3e-cb0c644ffd0e_characterizeImage_945756_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/1ca0a5c6-34b7-44b7-ae3e-cb0c644ffd0e_characterizeImage_945756_47.stderr parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/dd002729-4ef2-4369-a1e1-eb84677ccc36_isr_945756_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/dd002729-4ef2-4369-a1e1-eb84677ccc36_isr_945756_47.stderr parsl.dataflow.dflow INFO: Task 1056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2b7b2b63520688bf603ab6884767e0d with result from task 1056 parsl.dataflow.dflow DEBUG: Task 1056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1057 has memoization hash cbc95dfbbe035eb6c275aeefee7b3066 parsl.dataflow.memoization INFO: Task 1057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1057 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/0a0ac4ea-9417-4e33-8c95-327d95ba542b_characterizeImage_945755_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/0a0ac4ea-9417-4e33-8c95-327d95ba542b_characterizeImage_945755_47.stderr parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8dd92384-db5e-4192-8304-70055d7702e5_isr_945755_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8dd92384-db5e-4192-8304-70055d7702e5_isr_945755_47.stderr parsl.dataflow.dflow INFO: Task 1053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12e47533c66c3b10e766368c8709ac33 with result from task 1053 parsl.dataflow.dflow DEBUG: Task 1053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1054 has memoization hash fbac578086abc25fd7656b334060b8cc parsl.dataflow.memoization INFO: Task 1054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1054 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/95307f6d-0d54-4cdc-9058-07ee28ce879e_characterizeImage_945755_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/95307f6d-0d54-4cdc-9058-07ee28ce879e_characterizeImage_945755_30.stderr parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e8285f40-0fb5-44f0-8c98-d091ead19fd6_isr_945755_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e8285f40-0fb5-44f0-8c98-d091ead19fd6_isr_945755_30.stderr parsl.dataflow.dflow INFO: Task 1065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bccc1b119a9a558b238f60207295891e with result from task 1065 parsl.dataflow.dflow DEBUG: Task 1065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1066 has memoization hash b42522db37b12170c07ca559e32e5af3 parsl.dataflow.memoization INFO: Task 1066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1066 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/7583a6f2-13cd-4250-b2a3-5c6ef5ae3105_characterizeImage_945757_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/7583a6f2-13cd-4250-b2a3-5c6ef5ae3105_characterizeImage_945757_52.stderr parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/9db927fb-8650-4039-a61d-f7b6ef368daf_isr_945757_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/9db927fb-8650-4039-a61d-f7b6ef368daf_isr_945757_52.stderr parsl.dataflow.dflow INFO: Task 1080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b67205d7c0df8e89ccd58fdd890d49d with result from task 1080 parsl.dataflow.dflow DEBUG: Task 1080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1081 has memoization hash e3561240f1ef26fc7349559f82423b45 parsl.dataflow.memoization INFO: Task 1081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1081 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/4f641f87-7d29-4a14-b0bb-8ed823539c97_characterizeImage_945868_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/4f641f87-7d29-4a14-b0bb-8ed823539c97_characterizeImage_945868_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/e589c49c-4591-4689-aec0-8529d461232d_isr_945868_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/e589c49c-4591-4689-aec0-8529d461232d_isr_945868_9.stderr parsl.dataflow.dflow INFO: Task 1059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0387f513110c03badc345ffaa61cbc39 with result from task 1059 parsl.dataflow.dflow DEBUG: Task 1059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1060 has memoization hash 71cbfc696c74ba4087023dbc45b30494 parsl.dataflow.memoization INFO: Task 1060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1060 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4252745f-7c1a-4be4-8f57-84d027bf4d54_characterizeImage_945867_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4252745f-7c1a-4be4-8f57-84d027bf4d54_characterizeImage_945867_55.stderr parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8e2f3c72-d0d2-43b5-98f0-c7a5230341ce_isr_945867_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/8e2f3c72-d0d2-43b5-98f0-c7a5230341ce_isr_945867_55.stderr parsl.dataflow.dflow INFO: Task 1083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d0c1fdc340fa8b86fd031b956b70d66 with result from task 1083 parsl.dataflow.dflow DEBUG: Task 1083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1084 has memoization hash d4a9767bc736c66489405a3aec05ba60 parsl.dataflow.memoization INFO: Task 1084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1084 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/34685442-e428-4e5f-8158-ee9661a31b89_characterizeImage_945757_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/34685442-e428-4e5f-8158-ee9661a31b89_characterizeImage_945757_30.stderr parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/0148b963-3674-482f-8aed-4eb2d680a63d_isr_945757_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/0148b963-3674-482f-8aed-4eb2d680a63d_isr_945757_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:41 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6f1cad140ded797cc60dc73dccf5902 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash eaeb6d0504ebc6a2f5888a299aea2ec2 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/77faf695-7ba8-420c-80a2-88d3f7af4a06_calibrate_945757_14.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/77faf695-7ba8-420c-80a2-88d3f7af4a06_calibrate_945757_14.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/ddd3d83f-8dfc-4b6e-9eeb-b58e868de374_characterizeImage_945757_14.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/ddd3d83f-8dfc-4b6e-9eeb-b58e868de374_characterizeImage_945757_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:41 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc51b6a096aa02a0b07e69149019a44d with result from task 283 parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:42 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 610fa1b9306238208c69e8b7607f5f94 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246390360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/415c1617-bcbe-4b6c-a69b-82a311dab4c7_calibrate_945757_5.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/415c1617-bcbe-4b6c-a69b-82a311dab4c7_calibrate_945757_5.stderr parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/09eda586-77a6-44ef-8c34-b370f7a2f02d_characterizeImage_945757_5.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/09eda586-77a6-44ef-8c34-b370f7a2f02d_characterizeImage_945757_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:42 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57f662af6d9c8d9d2340b147337a8145 with result from task 885 parsl.dataflow.dflow DEBUG: Task 885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 886 has memoization hash 436b1be034659bf7661ae7c173301ec6 parsl.dataflow.memoization INFO: Task 886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461abf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/4f26ac23-2d58-476a-8d20-f572cd747cad_characterizeImage_945757_23.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/4f26ac23-2d58-476a-8d20-f572cd747cad_characterizeImage_945757_23.stderr parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a2130e19-25f9-4802-bfeb-7f4362088e02_isr_945757_23.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a2130e19-25f9-4802-bfeb-7f4362088e02_isr_945757_23.stderr parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 941e18dedb99df0bfc44fb651f0a3d7a with result from task 888 parsl.dataflow.dflow DEBUG: Task 888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 889 has memoization hash bf6cf1fc1016e2043253c8da5f69e9ab parsl.dataflow.memoization INFO: Task 889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/6863ad2e-2c66-45c3-a7fa-61a942f5a56c_characterizeImage_945868_45.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/6863ad2e-2c66-45c3-a7fa-61a942f5a56c_characterizeImage_945868_45.stderr parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c3f946d9-afcb-486e-9504-911f710b32a5_isr_945868_45.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c3f946d9-afcb-486e-9504-911f710b32a5_isr_945868_45.stderr parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2227d010bc92be36d37265b0e85c1789 with result from task 906 parsl.dataflow.dflow DEBUG: Task 906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 907 has memoization hash 038a56d01991780ae4ca2973124dc903 parsl.dataflow.memoization INFO: Task 907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246311f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/304715aa-b875-49fe-835d-63c46c0199f6_characterizeImage_945755_46.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/304715aa-b875-49fe-835d-63c46c0199f6_characterizeImage_945755_46.stderr parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b677757e-085e-4c9e-a746-d535d8543945_isr_945755_46.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/b677757e-085e-4c9e-a746-d535d8543945_isr_945755_46.stderr parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 503ce70a961253951d586155bd86187b with result from task 891 parsl.dataflow.dflow DEBUG: Task 891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 892 has memoization hash 03ae870062f2e6af0a03a291874580cd parsl.dataflow.memoization INFO: Task 892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a0d58a48-eacf-456b-960d-ba2c91e274b2_characterizeImage_945868_16.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a0d58a48-eacf-456b-960d-ba2c91e274b2_characterizeImage_945868_16.stderr parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/5f619edd-ff6e-46d1-9922-2e12e353cfb0_isr_945868_16.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/5f619edd-ff6e-46d1-9922-2e12e353cfb0_isr_945868_16.stderr parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e63a3422118ff58eb2dd51619669def0 with result from task 894 parsl.dataflow.dflow DEBUG: Task 894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 895 has memoization hash 6dd13a42cdf4520a65b12ee238e80ad3 parsl.dataflow.memoization INFO: Task 895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f50cbe57-8e97-4df4-81bf-7b32e36fe016_characterizeImage_945756_31.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f50cbe57-8e97-4df4-81bf-7b32e36fe016_characterizeImage_945756_31.stderr parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/44f719d3-092c-4958-9d23-91a2fe084b82_isr_945756_31.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/44f719d3-092c-4958-9d23-91a2fe084b82_isr_945756_31.stderr parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32a444da77d7672c0353cf5ae14090bf with result from task 900 parsl.dataflow.dflow DEBUG: Task 900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 901 has memoization hash a0a1bc030e2f9d2e644b785ac780169e parsl.dataflow.memoization INFO: Task 901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/7617e05d-5638-460a-96bf-b33bf1b2851a_characterizeImage_945757_10.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/7617e05d-5638-460a-96bf-b33bf1b2851a_characterizeImage_945757_10.stderr parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/e238fd75-56f1-4b74-aa74-908f5df8fa83_isr_945757_10.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/e238fd75-56f1-4b74-aa74-908f5df8fa83_isr_945757_10.stderr parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7f3922e123b382cca9aa54ce699bce5 with result from task 903 parsl.dataflow.dflow DEBUG: Task 903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 904 has memoization hash 93b9fd65d4a2cd441d921b96b0c5c5ed parsl.dataflow.memoization INFO: Task 904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a77e7a17-10bc-4a16-96a6-0d1c55956e88_characterizeImage_945757_12.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a77e7a17-10bc-4a16-96a6-0d1c55956e88_characterizeImage_945757_12.stderr parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/0972054f-d3c5-4662-b345-23ea2e3aeb4f_isr_945757_12.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/0972054f-d3c5-4662-b345-23ea2e3aeb4f_isr_945757_12.stderr parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f05490d65a202596dfa8eb676fbbfbb5 with result from task 897 parsl.dataflow.dflow DEBUG: Task 897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 898 has memoization hash 4e3bdb6917bd7738aa514c0a6d362728 parsl.dataflow.memoization INFO: Task 898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/9ba139c5-6e74-443c-b1aa-f83471c9dfc6_characterizeImage_945869_23.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/9ba139c5-6e74-443c-b1aa-f83471c9dfc6_characterizeImage_945869_23.stderr parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/992d5294-3547-4f25-9b1f-bd507e864d0b_isr_945869_23.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/992d5294-3547-4f25-9b1f-bd507e864d0b_isr_945869_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:43 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56476a9e04e6d7f7bc2d9e3a61f84796 with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash d5946eca98ad5c5d3435a15e6ffc4070 parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/472445ac-805e-4208-afdb-5a1a35f2cbe0_calibrate_945755_56.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/472445ac-805e-4208-afdb-5a1a35f2cbe0_calibrate_945755_56.stderr parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/df3c17b8-529c-4f12-b58c-d81aeafadefc_characterizeImage_945755_56.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/df3c17b8-529c-4f12-b58c-d81aeafadefc_characterizeImage_945755_56.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11e260a2aa7a9d2d0b2c51105772e889 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 7e41c2c20d6bdeb7cec45f14f2c7a55b parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247995120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/794551a9-77c5-42a5-b9e2-1181b6d3dfb0_calibrate_945757_26.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/794551a9-77c5-42a5-b9e2-1181b6d3dfb0_calibrate_945757_26.stderr parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/e063bd3a-90ff-428b-9d31-f8c3db13ee72_characterizeImage_945757_26.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/e063bd3a-90ff-428b-9d31-f8c3db13ee72_characterizeImage_945757_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:44 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:44 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:45 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef67b935115af5db7036e4c6d65d1f2c with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash dca41267b1afa9f1a6cef59749e23a47 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/230daac0-ec89-4ca8-8d83-c45b61aaa292_calibrate_945756_55.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/230daac0-ec89-4ca8-8d83-c45b61aaa292_calibrate_945756_55.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/dc240790-8a43-40e0-931b-22414e4c3038_characterizeImage_945756_55.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/dc240790-8a43-40e0-931b-22414e4c3038_characterizeImage_945756_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:45 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb94994bad1a969dfc15040e01ffe2a8 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 4abb4a3c0b7bb61db3a922d6964837f8 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c8ed4626-7e55-402e-890d-a1435ffe87c1_calibrate_945756_60.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c8ed4626-7e55-402e-890d-a1435ffe87c1_calibrate_945756_60.stderr parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/cd0f69e2-4f61-4c4a-9117-d292bae37df4_characterizeImage_945756_60.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/cd0f69e2-4f61-4c4a-9117-d292bae37df4_characterizeImage_945756_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:46 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:46 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1e5e7360889a791c2f6aec35ee24690 with result from task 1107 parsl.dataflow.dflow DEBUG: Task 1107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash 7173c1319e07c660b3337cb296285c07 parsl.dataflow.memoization INFO: Task 1108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460be5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1108 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/aea1a3ec-05b7-496d-a922-6dae7b57222f_characterizeImage_945757_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/aea1a3ec-05b7-496d-a922-6dae7b57222f_characterizeImage_945757_2.stderr parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/7e89b5b9-df3a-438c-b026-df5d10898b60_isr_945757_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/7e89b5b9-df3a-438c-b026-df5d10898b60_isr_945757_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:47 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee922c93e177a06e53d2c83c431871e2 with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 5ccaa3eff0e543a397b28815f0a63a36 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246391c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/bd31cb5c-d399-4b37-b42b-1eb5b0295d29_calibrate_945756_38.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/bd31cb5c-d399-4b37-b42b-1eb5b0295d29_calibrate_945756_38.stderr parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/8af5bcc5-691b-41ea-bdd3-ac80c7c49832_characterizeImage_945756_38.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/8af5bcc5-691b-41ea-bdd3-ac80c7c49832_characterizeImage_945756_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:48 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bd56bb684f1fa48f82c9a25a6901a17 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash fb794071255e50e53307ad1fa35d1d9a parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463909a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4e1a3d47-012f-415e-b712-67212a15891b_calibrate_945757_57.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4e1a3d47-012f-415e-b712-67212a15891b_calibrate_945757_57.stderr parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/6d35d883-8bd0-4744-9a2f-9dea1ce6935c_characterizeImage_945757_57.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/6d35d883-8bd0-4744-9a2f-9dea1ce6935c_characterizeImage_945757_57.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afa3eaa46d4b190425a412f2c775db87 with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash b4fbd9991d67a65b72b99cb868facf8b parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246392f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7556ee3f-13d8-443d-a224-e8939be15941_calibrate_945755_36.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7556ee3f-13d8-443d-a224-e8939be15941_calibrate_945755_36.stderr parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/f005b650-b0a4-4db2-a4b0-a0e9c1ff55b8_characterizeImage_945755_36.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/f005b650-b0a4-4db2-a4b0-a0e9c1ff55b8_characterizeImage_945755_36.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59ec6bc0193abc729608d9818242d9a3 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 6eee79fb1eae26402ed57172610a567e parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/dacf3ba3-4a84-4ebf-87e4-689ebb7e0b52_calibrate_945757_59.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/dacf3ba3-4a84-4ebf-87e4-689ebb7e0b52_calibrate_945757_59.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dadc18d9-e0c1-4d79-a580-b0b780d062b8_characterizeImage_945757_59.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dadc18d9-e0c1-4d79-a580-b0b780d062b8_characterizeImage_945757_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:48 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a68e11777a31eef4786ab29761ad3bd6 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 4c20fad654cb8cecd590d93e4df79f50 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/6e13b681-ed1c-4a2f-bb1d-4918ed335844_calibrate_945755_15.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/6e13b681-ed1c-4a2f-bb1d-4918ed335844_calibrate_945755_15.stderr parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/1b846dea-12fb-4aa1-82fa-0795f27ef107_characterizeImage_945755_15.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/1b846dea-12fb-4aa1-82fa-0795f27ef107_characterizeImage_945755_15.stderr parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e773729eb2060b81b983b05829ca48d2 with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash 72578fc0965c9f804eafffb4309ffbd5 parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/50f4c3ce-5ed6-4fc7-809b-3e0a9a3e2cf2_characterizeImage_945868_10.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/50f4c3ce-5ed6-4fc7-809b-3e0a9a3e2cf2_characterizeImage_945868_10.stderr parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/66cdd3b2-693b-4bab-ab1d-df1a8b30f66b_isr_945868_10.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/66cdd3b2-693b-4bab-ab1d-df1a8b30f66b_isr_945868_10.stderr parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42c0ebb6ea30b62d2cba2262df6cbc2f with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 7e6ef66b78bd1e549e4b946fe422cfcd parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246284f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6a945e68-db3b-4bbf-a28d-1b7c2a4aecd6_characterizeImage_945756_8.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6a945e68-db3b-4bbf-a28d-1b7c2a4aecd6_characterizeImage_945756_8.stderr parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/59a979ad-03ad-46eb-995f-4d1864699ca1_isr_945756_8.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/59a979ad-03ad-46eb-995f-4d1864699ca1_isr_945756_8.stderr parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 159b7237a592e315b46af9dbe4942d6b with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash fe9d0c4f9ba5241e512a3e750b24d75b parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246284c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/8d6da76a-992f-4a7c-9cd8-c75ab94f85ff_characterizeImage_945869_45.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/8d6da76a-992f-4a7c-9cd8-c75ab94f85ff_characterizeImage_945869_45.stderr parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/480e777a-a5a3-4538-9132-f65ae41fb23d_isr_945869_45.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/480e777a-a5a3-4538-9132-f65ae41fb23d_isr_945869_45.stderr parsl.dataflow.dflow INFO: Task 1110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96e2758d965d4ece6e9684fdbd90e6e0 with result from task 1110 parsl.dataflow.dflow DEBUG: Task 1110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1111 has memoization hash b2e8c631106202c337f86074b84ca300 parsl.dataflow.memoization INFO: Task 1111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460be660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1111 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/4673b9a6-9a17-462b-bab9-6ee730087fb4_characterizeImage_945755_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/4673b9a6-9a17-462b-bab9-6ee730087fb4_characterizeImage_945755_11.stderr parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/49312e09-1884-47a8-8b7c-7986c1e33a90_isr_945755_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/49312e09-1884-47a8-8b7c-7986c1e33a90_isr_945755_11.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed9a6c6238ed3e61f55991df1a10bbb5 with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash 5b37f349446e3d84742653f449645a6b parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246285260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/add1eb9c-c90f-496e-9a55-95c0aa91a8f6_characterizeImage_945869_6.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/add1eb9c-c90f-496e-9a55-95c0aa91a8f6_characterizeImage_945869_6.stderr parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/9bcedbd0-bc83-464e-816b-55e4ee76a180_isr_945869_6.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/9bcedbd0-bc83-464e-816b-55e4ee76a180_isr_945869_6.stderr parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e89360b6e63ece8e7f98d1324cd5eaf9 with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash a70f9ac26a109b60c561e8aa77b8d2fd parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/f44918c0-0de1-430e-aa7d-dad94a11ef8c_characterizeImage_945868_26.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/f44918c0-0de1-430e-aa7d-dad94a11ef8c_characterizeImage_945868_26.stderr parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8f430c6f-270d-46b9-95e3-015b6717eded_isr_945868_26.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/8f430c6f-270d-46b9-95e3-015b6717eded_isr_945868_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:49 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24fb349e980839d72397f56521b71a73 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 94d46d08ffd08ac652947a65be9bfb33 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479960c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/643e846d-9b3d-460d-be18-ad59ed6df5d4_calibrate_945757_35.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/643e846d-9b3d-460d-be18-ad59ed6df5d4_calibrate_945757_35.stderr parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b9e482da-6de9-49c2-a42c-e19fc2f096d0_characterizeImage_945757_35.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b9e482da-6de9-49c2-a42c-e19fc2f096d0_characterizeImage_945757_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:49 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4990d27d526e7fcc9c9a96cea686f63 with result from task 1113 parsl.dataflow.dflow DEBUG: Task 1113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash b809ac1148d8a6a7aec879df83a5884f parsl.dataflow.memoization INFO: Task 1114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460be8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1114 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2502c862-aa77-427c-bc14-a75a9ee3e910_characterizeImage_945867_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2502c862-aa77-427c-bc14-a75a9ee3e910_characterizeImage_945867_50.stderr parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4fb30e2e-343e-477b-b1b5-1fd00d153bfc_isr_945867_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4fb30e2e-343e-477b-b1b5-1fd00d153bfc_isr_945867_50.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddb0b41a1a7fec1a529d646bea248503 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash d2d84d82bd259f211e0931b9340fc7d2 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/2b3955b6-065b-44fc-b4f5-d51eddfab9db_calibrate_945757_55.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/2b3955b6-065b-44fc-b4f5-d51eddfab9db_calibrate_945757_55.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/71f431f0-a1ed-42b2-9b58-8ba15910cc7a_characterizeImage_945757_55.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/71f431f0-a1ed-42b2-9b58-8ba15910cc7a_characterizeImage_945757_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:50 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f2b01cdccd5cc07e89d7b7cfba16af0 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 93afe3441f2b26968de04b53ceaaaffe parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247994ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/f53899a3-33db-46f3-8402-aa9283292a43_calibrate_945757_41.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/f53899a3-33db-46f3-8402-aa9283292a43_calibrate_945757_41.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/4a42f0e2-15bf-4839-946c-385b601327a2_characterizeImage_945757_41.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/4a42f0e2-15bf-4839-946c-385b601327a2_characterizeImage_945757_41.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89f62d2afc0b005a28372c6589a95a8b with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 629d921aee606d83c41eed77fac244fd parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246284900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/fcd7a9a7-8b9a-44ff-bd3f-2888d8877434_characterizeImage_945757_9.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/fcd7a9a7-8b9a-44ff-bd3f-2888d8877434_characterizeImage_945757_9.stderr parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/460cfe67-c4af-4320-a996-5ab83531d2fc_isr_945757_9.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/460cfe67-c4af-4320-a996-5ab83531d2fc_isr_945757_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:50 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4046f29317ec3c3ce9ca8f5eedd387ec with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash 482f9c6565adc8eafef413d7699b39e3 parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462865c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c52199d5-b7f0-40d8-bf16-0e06de5ec3d2_calibrate_945756_30.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c52199d5-b7f0-40d8-bf16-0e06de5ec3d2_calibrate_945756_30.stderr parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b580f608-e6eb-4bc5-899a-8c696cdb6358_characterizeImage_945756_30.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b580f608-e6eb-4bc5-899a-8c696cdb6358_characterizeImage_945756_30.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dd48043e3147debc44d48280e28a68d with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash d539fb9042047df36760a9111053fce4 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247995760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/f99242b3-5713-4352-9e53-dcad61ba6ad0_calibrate_945757_17.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/f99242b3-5713-4352-9e53-dcad61ba6ad0_calibrate_945757_17.stderr parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/140c153b-7d22-4435-8447-797d8d5e2c6b_characterizeImage_945757_17.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/140c153b-7d22-4435-8447-797d8d5e2c6b_characterizeImage_945757_17.stderr parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10aa0edd61cab12087f016e97c5e738a with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash 40bcbe886b31243b7cd736b75f4ef3e9 parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462be840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/af125865-e34a-4a76-a7e2-a9ebe50f464e_calibrate_945755_27.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/af125865-e34a-4a76-a7e2-a9ebe50f464e_calibrate_945755_27.stderr parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/7f0d345f-e995-41ca-82e2-6476868dc21b_characterizeImage_945755_27.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/7f0d345f-e995-41ca-82e2-6476868dc21b_characterizeImage_945755_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:51 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:51 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f930603d1d898e97b4d42d69aba8a059 with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash 62e0795290e460b0c7f78d63e701e40f parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246311260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/36409f61-e809-49d0-92b3-3d61f7b39787_calibrate_945757_6.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/36409f61-e809-49d0-92b3-3d61f7b39787_calibrate_945757_6.stderr parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b8121dbb-b777-485f-8167-c011549760db_characterizeImage_945757_6.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b8121dbb-b777-485f-8167-c011549760db_characterizeImage_945757_6.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 613 try 1 failed parsl.dataflow.dflow ERROR: Task 613 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 06083405b98d22de019a3c2470e3379c with result from task 613 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 614 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 614 try 0 failed parsl.dataflow.dflow INFO: Task 614 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/3242494b-35ab-4b23-a215-5149ecb4e470_calibrate_945756_2.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/3242494b-35ab-4b23-a215-5149ecb4e470_calibrate_945756_2.stderr parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/5bc8eb4e-55c8-4502-b1e8-192cc0b1cd64_characterizeImage_945756_2.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/5bc8eb4e-55c8-4502-b1e8-192cc0b1cd64_characterizeImage_945756_2.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e75d15b94dc422c87ba00f6e837893a3 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 4132a3553dfa8164734d285d1f154c9a parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247996700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/4a7c5d2a-b089-49fe-a1f8-7e5afc96fa95_calibrate_945755_62.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/4a7c5d2a-b089-49fe-a1f8-7e5afc96fa95_calibrate_945755_62.stderr parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/d6e3260b-8f24-4b07-911f-8dc02222f6cc_characterizeImage_945755_62.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/d6e3260b-8f24-4b07-911f-8dc02222f6cc_characterizeImage_945755_62.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 142 try 1 failed parsl.dataflow.dflow ERROR: Task 142 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ab4632a424ad3db76f806f61db48d96a with result from task 142 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 143 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 143 try 0 failed parsl.dataflow.dflow INFO: Task 143 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/46e5b088-31ce-49b7-a471-eee91bba808e_calibrate_945867_2.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/46e5b088-31ce-49b7-a471-eee91bba808e_calibrate_945867_2.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2f3b6492-8334-4710-8723-08ee991967ae_characterizeImage_945867_2.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2f3b6492-8334-4710-8723-08ee991967ae_characterizeImage_945867_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:52 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56d58f88e163720abdd4a061bf8fad81 with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash 908dd02ad8cba23ae499161e860c99c2 parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462845e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/32dbcdfd-02c8-4175-9d5b-1870ba78a03b_characterizeImage_945757_31.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/32dbcdfd-02c8-4175-9d5b-1870ba78a03b_characterizeImage_945757_31.stderr parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/468d0b31-3e4c-4959-b51b-20b43a321130_isr_945757_31.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/468d0b31-3e4c-4959-b51b-20b43a321130_isr_945757_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:53 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 295ac273a5a2c5680a7838057949db83 with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash f576e43c2e0e9d807c6aa91822f77141 parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463107c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/9b73461e-d2b8-49e7-8407-bbec1785b187_characterizeImage_945755_61.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/9b73461e-d2b8-49e7-8407-bbec1785b187_characterizeImage_945755_61.stderr parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d2684aa5-1c2f-4292-91de-a0aa9ae4582e_isr_945755_61.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/d2684aa5-1c2f-4292-91de-a0aa9ae4582e_isr_945755_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 49 try 1 failed parsl.dataflow.dflow ERROR: Task 49 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9b296c3265ac177d46ab3db73e164dd7 with result from task 49 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 50 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 50 try 0 failed parsl.dataflow.dflow INFO: Task 50 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/96c86a44-9401-477c-a141-e6305da8dddb_calibrate_945755_2.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/96c86a44-9401-477c-a141-e6305da8dddb_calibrate_945755_2.stderr parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/699acbed-ec41-4e6a-a585-6a294ce9b24f_characterizeImage_945755_2.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/699acbed-ec41-4e6a-a585-6a294ce9b24f_characterizeImage_945755_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:53 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 763 try 0 failed parsl.dataflow.dflow INFO: Task 763 marked for retry parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0c89c7fe-5d97-460b-bcc1-9df1c4a37131_characterizeImage_945868_2.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0c89c7fe-5d97-460b-bcc1-9df1c4a37131_characterizeImage_945868_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash 0b886966824120c00b9e96b1c8652bd7 parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246313600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 763 try 1 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0c89c7fe-5d97-460b-bcc1-9df1c4a37131_characterizeImage_945868_2.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0c89c7fe-5d97-460b-bcc1-9df1c4a37131_characterizeImage_945868_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:54 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:54 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:55 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71d961a31eed8c962d5348606a737099 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 315182a2d8da5f6a2ae0ad6385aacb3e parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/c97f2829-28eb-4ee4-a30a-e67f50c4ee99_calibrate_945757_32.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/c97f2829-28eb-4ee4-a30a-e67f50c4ee99_calibrate_945757_32.stderr parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/571a2963-a91e-438c-aa6f-567dcc9eb220_characterizeImage_945757_32.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/571a2963-a91e-438c-aa6f-567dcc9eb220_characterizeImage_945757_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:55 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28136ca1001ca174e931d3c45fb4d77b with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash e15c9d434a5c65cfb7f631d9f9d4f334 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/514125a3-435d-49f3-9eac-5c875a809add_calibrate_945756_5.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/514125a3-435d-49f3-9eac-5c875a809add_calibrate_945756_5.stderr parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b4dbbe53-9382-4a64-8d0e-0dd101dfc7e3_characterizeImage_945756_5.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b4dbbe53-9382-4a64-8d0e-0dd101dfc7e3_characterizeImage_945756_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:56 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry faafa7835d1449b574a99785850f838c with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash fd798ef359f9c6269a819a817f045d1d parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/a6a8d124-98b7-4bac-b266-7818af939257_calibrate_945757_50.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/a6a8d124-98b7-4bac-b266-7818af939257_calibrate_945757_50.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/1ac5cc67-619f-4e0e-b294-b7fe79981361_characterizeImage_945757_50.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/1ac5cc67-619f-4e0e-b294-b7fe79981361_characterizeImage_945757_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:56 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:57 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1108 try 0 failed parsl.dataflow.dflow INFO: Task 1108 marked for retry parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/aea1a3ec-05b7-496d-a922-6dae7b57222f_characterizeImage_945757_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/aea1a3ec-05b7-496d-a922-6dae7b57222f_characterizeImage_945757_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash 7173c1319e07c660b3337cb296285c07 parsl.dataflow.memoization INFO: Task 1108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460be5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1108 try 1 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/aea1a3ec-05b7-496d-a922-6dae7b57222f_characterizeImage_945757_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/aea1a3ec-05b7-496d-a922-6dae7b57222f_characterizeImage_945757_2.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce0a94af5f758105211abcbf5f98e662 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 77d4371925aa6070414f30913ab32d1e parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/e95f3031-c1c6-4643-96a6-27e914816fc3_calibrate_945757_56.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/e95f3031-c1c6-4643-96a6-27e914816fc3_calibrate_945757_56.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/9be45a17-d518-4b33-9712-6c19764a7247_characterizeImage_945757_56.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/9be45a17-d518-4b33-9712-6c19764a7247_characterizeImage_945757_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:58 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:58 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:59 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:35:59 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:00 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 943fa85287bffde6237d183e4aa4842e with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 6a4fe2956a8216ac5a21cd390dea4955 parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bf4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/5098e02b-7b60-4ee9-8ff4-36d68f341163_calibrate_945757_46.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/5098e02b-7b60-4ee9-8ff4-36d68f341163_calibrate_945757_46.stderr parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/21a84baa-2f5c-474a-9df6-d555b46996a5_characterizeImage_945757_46.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/21a84baa-2f5c-474a-9df6-d555b46996a5_characterizeImage_945757_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:00 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a51c05554f557bf44f3ba3a61c6b6991 with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash 8c253a36d347688fc8282e2d3831e931 parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/ca9ee91b-a171-4669-8b31-11e719648560_calibrate_945755_5.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/ca9ee91b-a171-4669-8b31-11e719648560_calibrate_945755_5.stderr parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/929a82bc-ab10-4c8a-ab66-3caca7ae17dd_characterizeImage_945755_5.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/929a82bc-ab10-4c8a-ab66-3caca7ae17dd_characterizeImage_945755_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:01 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:02 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9678285b0456bfff96cb74197336f16c with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 024e5509c664d48fe302a8481ebf1266 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/48c07312-8790-4dae-bbbc-6ae0db432964_calibrate_945867_27.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/48c07312-8790-4dae-bbbc-6ae0db432964_calibrate_945867_27.stderr parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/f765694f-cd94-4cce-bedf-3a21ea29e1eb_characterizeImage_945867_27.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/f765694f-cd94-4cce-bedf-3a21ea29e1eb_characterizeImage_945867_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:02 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9542c163eebea7e70ba1a26ce3955bd8 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 9665dbb01316a4920d1a9b41ddccdd8f parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4dec484a-5e5c-47e8-88b9-0be890298a82_calibrate_945757_40.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4dec484a-5e5c-47e8-88b9-0be890298a82_calibrate_945757_40.stderr parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/d1cc340e-fac3-4cae-ac06-5cb3da00b59d_characterizeImage_945757_40.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/d1cc340e-fac3-4cae-ac06-5cb3da00b59d_characterizeImage_945757_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:03 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd54833672d495ef22bfb0b47b27eded with result from task 856 parsl.dataflow.dflow DEBUG: Task 856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 857 has memoization hash bba07fa05c518b04f91afbb7b680eec5 parsl.dataflow.memoization INFO: Task 857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/bebf7275-802f-4ee4-bf0b-f5e609f0faef_calibrate_945755_14.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/bebf7275-802f-4ee4-bf0b-f5e609f0faef_calibrate_945755_14.stderr parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/a816fa1a-f416-4e44-92ba-02c024a0d1bb_characterizeImage_945755_14.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/a816fa1a-f416-4e44-92ba-02c024a0d1bb_characterizeImage_945755_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:03 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47595e3f957c429e05dd61a0711ff577 with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash d063f982118832ad3ae305888e2b4665 parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462842c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/f0e2a9a6-bfd2-4685-9c37-cf38baa51fc8_characterizeImage_945755_49.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/f0e2a9a6-bfd2-4685-9c37-cf38baa51fc8_characterizeImage_945755_49.stderr parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/f69bf965-ec06-4c78-bbe0-fe6ec17bc80a_isr_945755_49.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/f69bf965-ec06-4c78-bbe0-fe6ec17bc80a_isr_945755_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:04 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73c6972eefb820e04974f6089dfe730b with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash 0e93c8127ede954d5d2fcdbcc8733938 parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/010cf681-91da-468d-9dc0-069aaeb81d25_calibrate_945757_38.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/010cf681-91da-468d-9dc0-069aaeb81d25_calibrate_945757_38.stderr parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/6b69b5fc-ed72-47a4-8b3e-8346d51be56d_characterizeImage_945757_38.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/6b69b5fc-ed72-47a4-8b3e-8346d51be56d_characterizeImage_945757_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:04 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 369 parsl.dataflow.strategy DEBUG: Executor multi has 369 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:36:05 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:36:05 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:06 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:06 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0f73d203e6d7ac3b83e82643bc11f36 with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash ccf031f3720148fd6cf1d49a78d7ee1b parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/d08eded8-f669-4e04-832c-49c01d3375a5_calibrate_945756_57.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/d08eded8-f669-4e04-832c-49c01d3375a5_calibrate_945756_57.stderr parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/aaab5ec8-8be2-4c83-b097-1c5e407ee40c_characterizeImage_945756_57.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/aaab5ec8-8be2-4c83-b097-1c5e407ee40c_characterizeImage_945756_57.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 503fae3b4233ddff03fb64b98438fc5a with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 1a51c84b146d597375c2eeee0e76f234 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246390cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/d6eed2e3-605a-4f06-9940-24c0557a5be5_calibrate_945868_50.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/d6eed2e3-605a-4f06-9940-24c0557a5be5_calibrate_945868_50.stderr parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/2002195d-9d6b-4ab2-8c3a-f15887be3e2a_characterizeImage_945868_50.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/2002195d-9d6b-4ab2-8c3a-f15887be3e2a_characterizeImage_945868_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:07 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9a5f7e29a9006d7ab894098c77ba941 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 1126d732a4e13d245aa5ee9aeefabdc6 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/59aaa762-6f8b-4b80-ad19-99050ba2ffa3_calibrate_945869_27.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/59aaa762-6f8b-4b80-ad19-99050ba2ffa3_calibrate_945869_27.stderr parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/69615b87-5512-47a0-a45a-56852204776f_characterizeImage_945869_27.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/69615b87-5512-47a0-a45a-56852204776f_characterizeImage_945869_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:07 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:08 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6262754a6cf900527e1cc26b2215027 with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash 8ae76d038e081481712d88c791530f42 parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/8803601b-9101-4b2e-9a28-9409d086c601_calibrate_945755_21.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/8803601b-9101-4b2e-9a28-9409d086c601_calibrate_945755_21.stderr parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/72164dd7-a3c8-40f8-b585-9748cfa8434b_characterizeImage_945755_21.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/72164dd7-a3c8-40f8-b585-9748cfa8434b_characterizeImage_945755_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:08 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82a09c7a821b4dc4d9f4cc49c3aade2e with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 6c62b81ca879d55e225b3379b6dd3374 parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d2c88b85-0c66-409c-9c47-1421641012b5_calibrate_945867_13.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d2c88b85-0c66-409c-9c47-1421641012b5_calibrate_945867_13.stderr parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4830841d-90f4-4bfa-a89e-74196b499a38_characterizeImage_945867_13.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4830841d-90f4-4bfa-a89e-74196b499a38_characterizeImage_945867_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:09 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3d5662fcf42535723b4aac3f04e0821 with result from task 874 parsl.dataflow.dflow DEBUG: Task 874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 875 has memoization hash 384f3414f38d26306685e305e720dae4 parsl.dataflow.memoization INFO: Task 875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461ab100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/48f21c4a-2052-49e9-8ce3-08f3938d6fe2_calibrate_945755_8.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/48f21c4a-2052-49e9-8ce3-08f3938d6fe2_calibrate_945755_8.stderr parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/bd61a087-739f-42af-ba27-860ab0571b7d_characterizeImage_945755_8.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/bd61a087-739f-42af-ba27-860ab0571b7d_characterizeImage_945755_8.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc83c07ae072f52f344388d1719d22a7 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 95e976bc5a08298a26dab4d09c5e34f7 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/e030b1d7-6014-4c6b-bf3d-af0efb83e2aa_calibrate_945869_51.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/e030b1d7-6014-4c6b-bf3d-af0efb83e2aa_calibrate_945869_51.stderr parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bdd4308d-5eab-4222-8a5d-3382a38f5b26_characterizeImage_945869_51.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bdd4308d-5eab-4222-8a5d-3382a38f5b26_characterizeImage_945869_51.stderr parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c47dce5c6e340ebf50605f31f1aa97e0 with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash 860af77e5084ebaa4b11f99818feaff3 parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/3afa57bb-fc28-4fb0-a3db-22adebf9a9cf_calibrate_945867_20.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/3afa57bb-fc28-4fb0-a3db-22adebf9a9cf_calibrate_945867_20.stderr parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/50ca3dfd-051d-4e1b-bf18-aa0c87c6e448_characterizeImage_945867_20.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/50ca3dfd-051d-4e1b-bf18-aa0c87c6e448_characterizeImage_945867_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:09 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:10 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3eb31087554ccbb5fbb2d03b2c7c8bc1 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 8e2235b950546ddb754ec5b3f774e0c0 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/cd51cab8-1b4f-4af8-8ef4-5efecec640d3_calibrate_945869_9.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/cd51cab8-1b4f-4af8-8ef4-5efecec640d3_calibrate_945869_9.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/e3ceb59e-fc1a-4fab-806e-7f794cc06c3f_characterizeImage_945869_9.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/e3ceb59e-fc1a-4fab-806e-7f794cc06c3f_characterizeImage_945869_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:11 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:11 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8b128389584cdbf25a453c06a75f4b7 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash e3dfce8eb6e57a3d399368e6d49480ff parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/45a470f3-0ffb-4a86-b6b4-84c8f4853bc3_calibrate_945757_49.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/45a470f3-0ffb-4a86-b6b4-84c8f4853bc3_calibrate_945757_49.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/823f1ba0-fc36-44fc-846c-56831b3ed019_characterizeImage_945757_49.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/823f1ba0-fc36-44fc-846c-56831b3ed019_characterizeImage_945757_49.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 250fb955c66fe22e2dc22c7fc586c2dd with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash acc19be14496fb63eb02665da66d0745 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2fbf43fd-26c3-447c-917c-aed553a6f9e8_calibrate_945868_27.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2fbf43fd-26c3-447c-917c-aed553a6f9e8_calibrate_945868_27.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0d81253e-8a90-4a1a-a098-f3c6867dc7ca_characterizeImage_945868_27.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0d81253e-8a90-4a1a-a098-f3c6867dc7ca_characterizeImage_945868_27.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1722371e49775958a89ae6885d00b13 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash fdf0a8ba71d82026282ef5046f997166 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/7de76ff1-d91a-4dab-8dfa-4f6e56799f92_calibrate_945867_37.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/7de76ff1-d91a-4dab-8dfa-4f6e56799f92_calibrate_945867_37.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/5529da29-41df-465c-b6e8-fcb0d8064e36_characterizeImage_945867_37.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/5529da29-41df-465c-b6e8-fcb0d8064e36_characterizeImage_945867_37.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c5b2a26ea3ee2a5d4b3e1f628825369 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f701bd8d-d39f-4f4e-b771-3cde81de548a_calibrate_945755_52.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f701bd8d-d39f-4f4e-b771-3cde81de548a_calibrate_945755_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:12 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:12 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 415 try 0 failed parsl.dataflow.dflow INFO: Task 415 marked for retry parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/8d904e3f-87f0-4602-b5f5-7c9327a328bf_characterizeImage_945757_61.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/8d904e3f-87f0-4602-b5f5-7c9327a328bf_characterizeImage_945757_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 08e0326b6388cd3bc6e195eb31b18119 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 1 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/8d904e3f-87f0-4602-b5f5-7c9327a328bf_characterizeImage_945757_61.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/8d904e3f-87f0-4602-b5f5-7c9327a328bf_characterizeImage_945757_61.stderr parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05016c5a6a4b6754b865f1c9e232839c with result from task 385 parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:13 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash c1f10264a14a7cc125b5ccda685066dd parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/39d5929c-f297-458b-9639-babc45f9378e_calibrate_945867_11.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/39d5929c-f297-458b-9639-babc45f9378e_calibrate_945867_11.stderr parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/6ee307cd-4429-46c4-9b27-4331d6cecaea_characterizeImage_945867_11.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/6ee307cd-4429-46c4-9b27-4331d6cecaea_characterizeImage_945867_11.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aea6fb08df13120c477fe4bf6aae5ee0 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 1fed1c0b25d94501aac9af387d6a8983 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/0de86e53-5f97-4887-a241-ac4db6aef5ed_calibrate_945756_54.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/0de86e53-5f97-4887-a241-ac4db6aef5ed_calibrate_945756_54.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/2077d2a9-3049-4890-aaee-6c24952558d2_characterizeImage_945756_54.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/2077d2a9-3049-4890-aaee-6c24952558d2_characterizeImage_945756_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:13 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:14 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4f92cd564f8e22ba21386246b4c69a2 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash eeef82c63f94b06d36c6e6fbac9e9cdb parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/0674f427-4cc2-4569-bb14-4cc4a7ac2432_calibrate_945867_48.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/0674f427-4cc2-4569-bb14-4cc4a7ac2432_calibrate_945867_48.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/04748ef6-0ea7-4396-9651-0812ae8b7036_characterizeImage_945867_48.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/04748ef6-0ea7-4396-9651-0812ae8b7036_characterizeImage_945867_48.stderr parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ca68e4ec037fdbdc0d27b14dde82c24 with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash 3d2b6da22a5b50deb11e1cec454ea9d4 parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246285bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/ddf97f43-5f7a-4806-a5a5-556ea0cf5b96_calibrate_945755_22.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/ddf97f43-5f7a-4806-a5a5-556ea0cf5b96_calibrate_945755_22.stderr parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/30acee16-a9db-4ab7-96a4-6b95d2934a2c_characterizeImage_945755_22.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/30acee16-a9db-4ab7-96a4-6b95d2934a2c_characterizeImage_945755_22.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 323623307ba2d258adaa8e8c5fc5d369 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/360b4176-df6e-40d0-a820-760ad7ecdc86_calibrate_945755_55.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/360b4176-df6e-40d0-a820-760ad7ecdc86_calibrate_945755_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:14 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8798fe99383f3e106920ca735599a4f6 with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash acfa9bb00fc3c53ac5ea4cdf14007903 parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246311e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/15f814c7-c3dc-49ef-8982-c37e7b31142a_calibrate_945869_33.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/15f814c7-c3dc-49ef-8982-c37e7b31142a_calibrate_945869_33.stderr parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/564be45a-e7bd-401d-93b7-91bb071613da_characterizeImage_945869_33.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/564be45a-e7bd-401d-93b7-91bb071613da_characterizeImage_945869_33.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a6b699338837e8cd544c85e7056f5c9 with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 8576e548ea099f9ebddf1c190f3a2c75 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/7037cf13-3e2c-4cfe-ac61-d71cc1424ae0_calibrate_945869_46.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/7037cf13-3e2c-4cfe-ac61-d71cc1424ae0_calibrate_945869_46.stderr parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/93204443-7c8d-406a-a4db-41e12492d800_characterizeImage_945869_46.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/93204443-7c8d-406a-a4db-41e12492d800_characterizeImage_945869_46.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1108 try 1 failed parsl.dataflow.dflow ERROR: Task 1108 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7173c1319e07c660b3337cb296285c07 with result from task 1108 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 1109 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 1109 try 0 failed parsl.dataflow.dflow INFO: Task 1109 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/09d83ac3-61e5-4582-956b-dfb998fe25e0_calibrate_945757_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/09d83ac3-61e5-4582-956b-dfb998fe25e0_calibrate_945757_2.stderr parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/aea1a3ec-05b7-496d-a922-6dae7b57222f_characterizeImage_945757_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/aea1a3ec-05b7-496d-a922-6dae7b57222f_characterizeImage_945757_2.stderr parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d666fdcdbada8651f5570bf9e80e559d with result from task 760 parsl.dataflow.dflow DEBUG: Task 760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 761 has memoization hash 42afbb2239ab8ea356575c58fe9249bc parsl.dataflow.memoization INFO: Task 761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246313240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/92744ff0-cef6-48c6-9b25-ba901fcdd950_calibrate_945756_59.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/92744ff0-cef6-48c6-9b25-ba901fcdd950_calibrate_945756_59.stderr parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/d6d4a07b-e781-4f94-b590-48b17e2dabcb_characterizeImage_945756_59.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/d6d4a07b-e781-4f94-b590-48b17e2dabcb_characterizeImage_945756_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:15 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:16 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 776f21bc713c93a51f6f8998dd65ab12 with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/964b792c-4e1f-4683-8c7f-c280b79dce14_calibrate_945756_46.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/964b792c-4e1f-4683-8c7f-c280b79dce14_calibrate_945756_46.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c2ec4040785683e5125a248e7654959 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 38521d12c90f5ec9fc7da7d9083cb34b parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/76aa4329-8ff3-44aa-9674-d4d79974b46f_calibrate_945869_48.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/76aa4329-8ff3-44aa-9674-d4d79974b46f_calibrate_945869_48.stderr parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1519bd68-81d2-48cc-86e5-9b0d00beccc2_characterizeImage_945869_48.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1519bd68-81d2-48cc-86e5-9b0d00beccc2_characterizeImage_945869_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:16 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdacfd17de94b32905e6d76d2fb4c248 with result from task 877 parsl.dataflow.dflow DEBUG: Task 877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 878 has memoization hash ec5af748ad530dd4dce3bbc13ee92b7f parsl.dataflow.memoization INFO: Task 878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461ab420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/932567e1-e46f-4ad1-8b49-98c6a0d843c9_calibrate_945755_28.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/932567e1-e46f-4ad1-8b49-98c6a0d843c9_calibrate_945755_28.stderr parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/6c779dfe-4c78-461e-a826-3ecfa67f8984_characterizeImage_945755_28.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/6c779dfe-4c78-461e-a826-3ecfa67f8984_characterizeImage_945755_28.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eae2ed32b43fe7d078fd8f8d942295c2 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash bf1687801b965460057ad36c46709ef1 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247995440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/3541ef42-41a7-46a7-bcb8-4e117ef62f0f_calibrate_945867_46.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/3541ef42-41a7-46a7-bcb8-4e117ef62f0f_calibrate_945867_46.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ea4537d9-328f-48bb-906b-cc877078c3da_characterizeImage_945867_46.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ea4537d9-328f-48bb-906b-cc877078c3da_characterizeImage_945867_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:17 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aa3a474c1c9528a2d9aad65af350949 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d2943037-aecd-44e2-b128-0d53f86ea310_calibrate_945757_39.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d2943037-aecd-44e2-b128-0d53f86ea310_calibrate_945757_39.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c175cab1b26e96c244a5dda892aed88 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 343c3b374000ea6e0bef68e17cd21686 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/b6be87e9-89a7-42c9-9db3-6cdfb5eb2095_calibrate_945755_59.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/b6be87e9-89a7-42c9-9db3-6cdfb5eb2095_calibrate_945755_59.stderr parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/2507f02a-bad6-4b16-8547-727a833ce23a_characterizeImage_945755_59.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/2507f02a-bad6-4b16-8547-727a833ce23a_characterizeImage_945755_59.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d300d4743bd85db25e2bb5741fff6053 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 87c296e038dd895f7ec76f88b9824ef3 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247917b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/fbbe104d-a8fb-4221-8e72-d113e81caf81_calibrate_945755_23.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/fbbe104d-a8fb-4221-8e72-d113e81caf81_calibrate_945755_23.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/22c98ded-882f-438a-a980-66982690d9d7_characterizeImage_945755_23.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/22c98ded-882f-438a-a980-66982690d9d7_characterizeImage_945755_23.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd183995066a1e19d78473f3d53fed52 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash ec7e63387329bce623fd3eca4da4af0d parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479947c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/6912b654-537f-4bf7-9659-bc957f6ab1c8_calibrate_945867_40.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/6912b654-537f-4bf7-9659-bc957f6ab1c8_calibrate_945867_40.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/6a9bfa78-3bad-40e3-945e-015c8196fe96_characterizeImage_945867_40.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/6a9bfa78-3bad-40e3-945e-015c8196fe96_characterizeImage_945867_40.stderr parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31b9e0f5ed720d8ea6008a45cd137a4f with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 82eae41a444f223349934896396df377 parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/b9343450-1db9-4316-91a5-e23053d13084_calibrate_945755_4.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/b9343450-1db9-4316-91a5-e23053d13084_calibrate_945755_4.stderr parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/dd5e5de6-f674-4331-b315-5d634dac905f_characterizeImage_945755_4.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/dd5e5de6-f674-4331-b315-5d634dac905f_characterizeImage_945755_4.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47b2afdda48fe787cbccc1abba4c2f6a with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/d99990e1-9ac7-4706-adc3-331bbb8140e0_calibrate_945756_26.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/d99990e1-9ac7-4706-adc3-331bbb8140e0_calibrate_945756_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:17 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e57adb3b4ea0ebe7209059579da10107 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 0eb539d49f52d49c57371f357efa29c7 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479979c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/1d43f2a4-eaf5-4df6-89a4-a0979bc08891_calibrate_945867_5.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/1d43f2a4-eaf5-4df6-89a4-a0979bc08891_calibrate_945867_5.stderr parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/1c5c6d88-0696-49a2-bfca-60feebe6aca4_characterizeImage_945867_5.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/1c5c6d88-0696-49a2-bfca-60feebe6aca4_characterizeImage_945867_5.stderr parsl.dataflow.dflow INFO: Task 1102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8049fa7d3679e690174992f6037ede1b with result from task 1102 parsl.dataflow.dflow DEBUG: Task 1102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1103 has memoization hash bf38ca7cf1823a54a813e0144b4bce18 parsl.dataflow.memoization INFO: Task 1103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1103 try 0 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/fd6f9235-b2b0-4510-9913-822b83a09c27_calibrate_945756_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/fd6f9235-b2b0-4510-9913-822b83a09c27_calibrate_945756_28.stderr parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ff5a127a-dda7-4da1-949d-674691b99557_characterizeImage_945756_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ff5a127a-dda7-4da1-949d-674691b99557_characterizeImage_945756_28.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54a81d3a4a76d84f73736653d9b2aee9 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash 3f769e81fd57c69afa4e86f49b5f346a parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/db8e32fc-643a-4541-9943-a6c418156c43_calibrate_945869_60.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/db8e32fc-643a-4541-9943-a6c418156c43_calibrate_945869_60.stderr parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/647d38e2-052f-4260-973a-7e2cef2db04b_characterizeImage_945869_60.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/647d38e2-052f-4260-973a-7e2cef2db04b_characterizeImage_945869_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:18 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52b6e8afb07771b107421b0d349b490a with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 911882864613e4db4f7f3ea023b817a1 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479174c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/a6bb827d-c7b6-4c35-8fdf-912d92bcf878_calibrate_945869_53.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/a6bb827d-c7b6-4c35-8fdf-912d92bcf878_calibrate_945869_53.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/3ba911be-f824-4723-b370-91a0b41e61a9_characterizeImage_945869_53.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/3ba911be-f824-4723-b370-91a0b41e61a9_characterizeImage_945869_53.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 008d81f8b4f4f83fd4cb377049612dc3 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 13285ea500519c1cc23ef5a1b32ceff5 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/2b4632d2-991d-4f5c-8d0f-7d583668e2ee_calibrate_945755_32.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/2b4632d2-991d-4f5c-8d0f-7d583668e2ee_calibrate_945755_32.stderr parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/6ff15723-3f8e-4a40-8916-2df8e32bdc03_characterizeImage_945755_32.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/6ff15723-3f8e-4a40-8916-2df8e32bdc03_characterizeImage_945755_32.stderr parsl.dataflow.dflow INFO: Task 1057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbc95dfbbe035eb6c275aeefee7b3066 with result from task 1057 parsl.dataflow.dflow DEBUG: Task 1057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1058 has memoization hash 4e9ef6a9f743d7ffdafd51ce5de39a30 parsl.dataflow.memoization INFO: Task 1058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1058 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/46048db4-e96d-4cfc-a540-7d5dae959db6_calibrate_945755_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/46048db4-e96d-4cfc-a540-7d5dae959db6_calibrate_945755_47.stderr parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/0a0ac4ea-9417-4e33-8c95-327d95ba542b_characterizeImage_945755_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/0a0ac4ea-9417-4e33-8c95-327d95ba542b_characterizeImage_945755_47.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b35df9d9cfe607960c9af6c3675e3641 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/ac0b9dcc-becb-4520-bdbf-b6db41dc809b_calibrate_945756_36.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/ac0b9dcc-becb-4520-bdbf-b6db41dc809b_calibrate_945756_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:18 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5498eca220284bae173bf62c6f00aa9 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash c95df3e760f27ee03f7251297e259b30 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/6070f19a-b36f-4f83-b1b6-62179096ff15_calibrate_945867_9.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/6070f19a-b36f-4f83-b1b6-62179096ff15_calibrate_945867_9.stderr parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/f8113798-c437-4bea-bdeb-ed6dc3167b70_characterizeImage_945867_9.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/f8113798-c437-4bea-bdeb-ed6dc3167b70_characterizeImage_945867_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:19 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 038a56d01991780ae4ca2973124dc903 with result from task 907 parsl.dataflow.dflow DEBUG: Task 907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 908 has memoization hash bc5cecd431e6bab42c50f7093c3fbcc3 parsl.dataflow.memoization INFO: Task 908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/08ad4f1f-611d-4433-8070-2b7f8b6cdd10_calibrate_945755_46.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/08ad4f1f-611d-4433-8070-2b7f8b6cdd10_calibrate_945755_46.stderr parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/304715aa-b875-49fe-835d-63c46c0199f6_characterizeImage_945755_46.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/304715aa-b875-49fe-835d-63c46c0199f6_characterizeImage_945755_46.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57a8ec3b0a1622eafd0aacbcec04c5b6 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/8cf38951-bcd9-4b5b-b27b-cf0a581b64f1_calibrate_945757_51.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/8cf38951-bcd9-4b5b-b27b-cf0a581b64f1_calibrate_945757_51.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3caece5066cdee40f257e36a6dceaaaf with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash fa14f9d638a47bdb84299018a8a7380c parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7ab84536-01bd-40be-874f-05012a579d5c_calibrate_945755_20.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7ab84536-01bd-40be-874f-05012a579d5c_calibrate_945755_20.stderr parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/d11060d0-735f-4c3d-b252-5d82c4de982a_characterizeImage_945755_20.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/d11060d0-735f-4c3d-b252-5d82c4de982a_characterizeImage_945755_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:19 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 763 try 1 failed parsl.dataflow.dflow ERROR: Task 763 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 0b886966824120c00b9e96b1c8652bd7 with result from task 763 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 764 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 764 try 0 failed parsl.dataflow.dflow INFO: Task 764 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/e787d29d-4192-44ac-a2af-32195fec74d3_calibrate_945868_2.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/e787d29d-4192-44ac-a2af-32195fec74d3_calibrate_945868_2.stderr parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0c89c7fe-5d97-460b-bcc1-9df1c4a37131_characterizeImage_945868_2.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0c89c7fe-5d97-460b-bcc1-9df1c4a37131_characterizeImage_945868_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:20 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ccaa3eff0e543a397b28815f0a63a36 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/bd31cb5c-d399-4b37-b42b-1eb5b0295d29_calibrate_945756_38.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/bd31cb5c-d399-4b37-b42b-1eb5b0295d29_calibrate_945756_38.stderr parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c7b0a483f6227edf2b40b4d5166b9d4 with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 4e7f00ba145af188c74d4396e4f41e0b parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/55433490-2383-4ca0-ad25-044b8bef9082_calibrate_945755_45.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/55433490-2383-4ca0-ad25-044b8bef9082_calibrate_945755_45.stderr parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/0dff9db8-cb1f-4975-ac61-3b1093211a43_characterizeImage_945755_45.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/0dff9db8-cb1f-4975-ac61-3b1093211a43_characterizeImage_945755_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:21 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c6747afc7a9b88db64bb1dabb3afa1c with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash d50a6c909f12b6e2fb8e55f6c402d341 parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/91870a96-6f47-4f2d-9918-f93bd76a5544_calibrate_945755_57.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/91870a96-6f47-4f2d-9918-f93bd76a5544_calibrate_945755_57.stderr parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/e3b678e3-d9fe-45c8-9660-6cdb8753dec1_characterizeImage_945755_57.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/e3b678e3-d9fe-45c8-9660-6cdb8753dec1_characterizeImage_945755_57.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fd8c1e952d7cb931a289f4484e211c9 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 33abfec5e5818a90ffb93ff4baabae68 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/902b7813-7713-45dc-8cd8-f0675abc093f_calibrate_945868_37.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/902b7813-7713-45dc-8cd8-f0675abc093f_calibrate_945868_37.stderr parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/b0abcd38-9053-4b22-8f17-1c590a9da560_characterizeImage_945868_37.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/b0abcd38-9053-4b22-8f17-1c590a9da560_characterizeImage_945868_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:21 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c3ffa49e1ccea0a760cd5fac505a87f with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash fe3aedaa1d6d6f9e1487a518c541ddfc parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246391940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 766 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/ad7a8f04-557f-4b7a-8195-c893f2de0e09_calibrate_945756_35.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/ad7a8f04-557f-4b7a-8195-c893f2de0e09_calibrate_945756_35.stderr parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ff8cffaa-a1ee-4ed3-b332-12a70070d9d3_characterizeImage_945756_35.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ff8cffaa-a1ee-4ed3-b332-12a70070d9d3_characterizeImage_945756_35.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3988e94414b91b149017674a9a901b0 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 295f7143b0314c8f0a86800a60bbd9db parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 767 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/4f8cee6f-3055-41cf-a752-4aa2358952a4_calibrate_945755_7.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/4f8cee6f-3055-41cf-a752-4aa2358952a4_calibrate_945755_7.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/a9e8f11c-ed4c-48e2-ad31-8746ccffc444_characterizeImage_945755_7.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/a9e8f11c-ed4c-48e2-ad31-8746ccffc444_characterizeImage_945755_7.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4fbd9991d67a65b72b99cb868facf8b with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7556ee3f-13d8-443d-a224-e8939be15941_calibrate_945755_36.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7556ee3f-13d8-443d-a224-e8939be15941_calibrate_945755_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:22 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 749afe1e692691ca647fdf193d0a4a50 with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 99a3f241ac90ad6d73782883b75547e1 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246391f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 768 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/93f61585-0b79-4725-845c-2be8dc40188c_calibrate_945868_53.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/93f61585-0b79-4725-845c-2be8dc40188c_calibrate_945868_53.stderr parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/340b3d66-bcaa-4656-9fc8-997468df3cf5_characterizeImage_945868_53.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/340b3d66-bcaa-4656-9fc8-997468df3cf5_characterizeImage_945868_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:23 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c40b76db24a47127af5ac1c749c7e41d with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash e363d7628c9880d2c8952426051c1d03 parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 769 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/b9645bcb-c7c8-4bbb-b123-d407ec7a9f19_calibrate_945756_22.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/b9645bcb-c7c8-4bbb-b123-d407ec7a9f19_calibrate_945756_22.stderr parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4d4b5088-6502-4984-9184-721152ed8fc6_characterizeImage_945756_22.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4d4b5088-6502-4984-9184-721152ed8fc6_characterizeImage_945756_22.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a9ad0a016edcaec86dc2397660da797 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 133d575763184a10bd54fe6db90180da parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 770 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/17ba65ad-a7fb-4610-9e8d-00d2253fde13_calibrate_945868_58.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/17ba65ad-a7fb-4610-9e8d-00d2253fde13_calibrate_945868_58.stderr parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/d7866d19-bd70-4ea3-8d1c-c093dadc1ead_characterizeImage_945868_58.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/d7866d19-bd70-4ea3-8d1c-c093dadc1ead_characterizeImage_945868_58.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b65fbfefe43affee4b1fd05ce20a257 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 78a968adbf7cb06419e3e404715cf1a9 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247994360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 771 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/40bab741-f770-407f-b5c4-8748c2ebe95b_calibrate_945869_10.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/40bab741-f770-407f-b5c4-8748c2ebe95b_calibrate_945869_10.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/d0248f33-9412-4e93-8dab-ce724eade571_characterizeImage_945869_10.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/d0248f33-9412-4e93-8dab-ce724eade571_characterizeImage_945869_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:23 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5946eca98ad5c5d3435a15e6ffc4070 with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/472445ac-805e-4208-afdb-5a1a35f2cbe0_calibrate_945755_56.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/472445ac-805e-4208-afdb-5a1a35f2cbe0_calibrate_945755_56.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d001cdd4f4750a709635e1675ff97383 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 3d2f0a14c0d2f442de2989829bfb0dcf parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 772 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/1149017f-6a2d-4a54-ab76-f19c3f4ad7a3_calibrate_945867_36.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/1149017f-6a2d-4a54-ab76-f19c3f4ad7a3_calibrate_945867_36.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/9caee69b-d1b6-4b2f-8e91-d8e482a6e859_characterizeImage_945867_36.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/9caee69b-d1b6-4b2f-8e91-d8e482a6e859_characterizeImage_945867_36.stderr parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39aa50c0686a56bace3f68ebb01160d3 with result from task 847 parsl.dataflow.dflow DEBUG: Task 847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 848 has memoization hash 450822c155d39f09d3913e986d630f5a parsl.dataflow.memoization INFO: Task 848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 773 parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b0d5ae4d-c5a3-4586-b234-57987e768b7f_calibrate_945869_47.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b0d5ae4d-c5a3-4586-b234-57987e768b7f_calibrate_945869_47.stderr parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/4ca5bb72-b1b5-49f2-a4aa-f4c8aa6e55d0_characterizeImage_945869_47.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/4ca5bb72-b1b5-49f2-a4aa-f4c8aa6e55d0_characterizeImage_945869_47.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97497bf69b536c480942b9cf234930d5 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/1fc09c80-ae30-48e1-af7a-c113f0ffc03b_calibrate_945757_22.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/1fc09c80-ae30-48e1-af7a-c113f0ffc03b_calibrate_945757_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:24 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:24 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:25 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d32703b5b24a3a657a490d435fdca047 with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 8aff4ee8e5580b21679a38d29d6590cd parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 774 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/599b0918-0c0c-4493-9bc0-bef6f18c0c29_calibrate_945756_56.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/599b0918-0c0c-4493-9bc0-bef6f18c0c29_calibrate_945756_56.stderr parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4a1dc1f8-9dec-44c9-afda-fe391247040e_characterizeImage_945756_56.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4a1dc1f8-9dec-44c9-afda-fe391247040e_characterizeImage_945756_56.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14b98f19e2da1bea94d95b40c263fa97 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 7e5e315cf9720c1cdc859137144ad69a parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 775 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7868f511-4798-4b4b-a256-dd9b6edb84cd_calibrate_945755_42.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7868f511-4798-4b4b-a256-dd9b6edb84cd_calibrate_945755_42.stderr parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/8596220f-13ef-4619-8160-dba51ba73132_characterizeImage_945755_42.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/8596220f-13ef-4619-8160-dba51ba73132_characterizeImage_945755_42.stderr parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 149de646a1f497701010fcf89285d57d with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash f299d525c84f98e4d4c666bc2f6d6750 parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246310cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 776 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/c6f3f557-e01d-4e30-a391-7bc480aaa7ad_calibrate_945755_26.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/c6f3f557-e01d-4e30-a391-7bc480aaa7ad_calibrate_945755_26.stderr parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/b58e0d25-249c-44fb-8734-7df777dab11d_characterizeImage_945755_26.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/b58e0d25-249c-44fb-8734-7df777dab11d_characterizeImage_945755_26.stderr parsl.dataflow.dflow INFO: Task 1090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eae6a8a7537f975606af42340f0af47a with result from task 1090 parsl.dataflow.dflow DEBUG: Task 1090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1091 has memoization hash 02c4767238506fda9210a4bb25b81f34 parsl.dataflow.memoization INFO: Task 1091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1091 try 0 launched on executor multi with executor id 777 parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/189d78e6-3e29-44ce-8efa-f3d9f2f21881_calibrate_945755_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/189d78e6-3e29-44ce-8efa-f3d9f2f21881_calibrate_945755_31.stderr parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/8e027e21-2443-4d11-840d-4867ec72841a_characterizeImage_945755_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/8e027e21-2443-4d11-840d-4867ec72841a_characterizeImage_945755_31.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a067d04b4d3622abe2365664c2e94ff with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 7ffb47421a1900e3327e440f4d0e4f87 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 778 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4404c18e-f6ec-454e-a7b5-a11516eaa5d2_calibrate_945757_62.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4404c18e-f6ec-454e-a7b5-a11516eaa5d2_calibrate_945757_62.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/077ab6b2-4070-4081-a6d0-55107de2176e_characterizeImage_945757_62.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/077ab6b2-4070-4081-a6d0-55107de2176e_characterizeImage_945757_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:25 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 712 try 0 failed parsl.dataflow.dflow INFO: Task 712 marked for retry parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/9b73461e-d2b8-49e7-8407-bbec1785b187_characterizeImage_945755_61.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/9b73461e-d2b8-49e7-8407-bbec1785b187_characterizeImage_945755_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash f576e43c2e0e9d807c6aa91822f77141 parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463107c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 1 launched on executor multi with executor id 779 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/9b73461e-d2b8-49e7-8407-bbec1785b187_characterizeImage_945755_61.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/9b73461e-d2b8-49e7-8407-bbec1785b187_characterizeImage_945755_61.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dca41267b1afa9f1a6cef59749e23a47 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/230daac0-ec89-4ca8-8d83-c45b61aaa292_calibrate_945756_55.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/230daac0-ec89-4ca8-8d83-c45b61aaa292_calibrate_945756_55.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f13f8f9cbdd26a74b712b603985950b with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash bfbeba4c6a1ed4f6e8a24690720face3 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 780 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/e3850802-3eff-44ec-8c1f-88bf917391b3_calibrate_945757_8.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/e3850802-3eff-44ec-8c1f-88bf917391b3_calibrate_945757_8.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/ddf7ee27-af2d-4363-80dd-85037b445efb_characterizeImage_945757_8.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/ddf7ee27-af2d-4363-80dd-85037b445efb_characterizeImage_945757_8.stderr parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 756cbb1b4e6e8c59508fe0ed452e96e4 with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash a4af508c2e047727aebb03654032afaf parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 781 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/10fdab73-3c13-4646-98c9-5eae55933037_calibrate_945869_34.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/10fdab73-3c13-4646-98c9-5eae55933037_calibrate_945869_34.stderr parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/68aa802c-962f-49f4-a85f-c8deff19a730_characterizeImage_945869_34.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/68aa802c-962f-49f4-a85f-c8deff19a730_characterizeImage_945869_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:26 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbef4b5b52e6eda873bbb75a1cacc1f7 with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 2dd3b8deed40204d58cd497aad30d350 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 782 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/73d7fe3a-3b95-4d7b-8ab8-dd0bb9f5eb17_calibrate_945756_50.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/73d7fe3a-3b95-4d7b-8ab8-dd0bb9f5eb17_calibrate_945756_50.stderr parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6202145e-5725-4b2a-8be9-3aef559b5886_characterizeImage_945756_50.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6202145e-5725-4b2a-8be9-3aef559b5886_characterizeImage_945756_50.stderr parsl.dataflow.dflow INFO: Task 1054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbac578086abc25fd7656b334060b8cc with result from task 1054 parsl.dataflow.dflow DEBUG: Task 1054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash 3dd12c154a48ba79abfae90c3f428ddd parsl.dataflow.memoization INFO: Task 1055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1055 try 0 launched on executor multi with executor id 783 parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/b3d2224b-65cc-48ff-a763-bedbff3f3600_calibrate_945755_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/b3d2224b-65cc-48ff-a763-bedbff3f3600_calibrate_945755_30.stderr parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/95307f6d-0d54-4cdc-9058-07ee28ce879e_characterizeImage_945755_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/95307f6d-0d54-4cdc-9058-07ee28ce879e_characterizeImage_945755_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:27 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 157 try 0 failed parsl.dataflow.dflow INFO: Task 157 marked for retry parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/109edcee-de71-4665-a0bb-9c8ade9f452d_characterizeImage_945756_61.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/109edcee-de71-4665-a0bb-9c8ade9f452d_characterizeImage_945756_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 45db7ac2a82f88bd80ff37bac26a2d4a parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 1 launched on executor multi with executor id 784 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/109edcee-de71-4665-a0bb-9c8ade9f452d_characterizeImage_945756_61.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/109edcee-de71-4665-a0bb-9c8ade9f452d_characterizeImage_945756_61.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d45bfde5a6f6285c3f71cd7585fdb33f with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 0f80f35937022ff053ec3fab1368f3d5 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 785 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2c853e0b-205e-485a-9230-c6bf696e332a_calibrate_945868_15.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2c853e0b-205e-485a-9230-c6bf696e332a_calibrate_945868_15.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/8f768533-dd79-4cc4-b3f0-d59f03de01ef_characterizeImage_945868_15.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/8f768533-dd79-4cc4-b3f0-d59f03de01ef_characterizeImage_945868_15.stderr parsl.dataflow.dflow INFO: Task 1087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8e03885fb5d100ba73952b593196684 with result from task 1087 parsl.dataflow.dflow DEBUG: Task 1087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1088 has memoization hash d3ff1e5e7571de64c3abe1eeeb2e9b54 parsl.dataflow.memoization INFO: Task 1088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1088 try 0 launched on executor multi with executor id 786 parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/4df201b2-cad8-4d32-8402-1826f1091a56_calibrate_945756_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/4df201b2-cad8-4d32-8402-1826f1091a56_calibrate_945756_47.stderr parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/1ca0a5c6-34b7-44b7-ae3e-cb0c644ffd0e_characterizeImage_945756_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/1ca0a5c6-34b7-44b7-ae3e-cb0c644ffd0e_characterizeImage_945756_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:27 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e598fe469cc91e599d5905e053d6b5e with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 06808e46c9ffe7226cd371adf483ea6f parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 787 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/659bd90d-a59d-473f-88e7-de0f2f76e0a0_calibrate_945869_41.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/659bd90d-a59d-473f-88e7-de0f2f76e0a0_calibrate_945869_41.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/89efe46e-bee3-4d1a-9c36-63b5c5346615_characterizeImage_945869_41.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/89efe46e-bee3-4d1a-9c36-63b5c5346615_characterizeImage_945869_41.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaeb6d0504ebc6a2f5888a299aea2ec2 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/77faf695-7ba8-420c-80a2-88d3f7af4a06_calibrate_945757_14.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/77faf695-7ba8-420c-80a2-88d3f7af4a06_calibrate_945757_14.stderr parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7d88b81a308147f337c539dbff027b0 with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 1bcbc766d20e20d5e4151c58a7bfb64f parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 788 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/b5a8dacd-c87c-4d06-922e-93a4895652e2_calibrate_945867_58.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/b5a8dacd-c87c-4d06-922e-93a4895652e2_calibrate_945867_58.stderr parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2dea0aa4-1dec-4060-9374-c19868103550_characterizeImage_945867_58.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2dea0aa4-1dec-4060-9374-c19868103550_characterizeImage_945867_58.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4abb4a3c0b7bb61db3a922d6964837f8 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c8ed4626-7e55-402e-890d-a1435ffe87c1_calibrate_945756_60.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c8ed4626-7e55-402e-890d-a1435ffe87c1_calibrate_945756_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:28 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59b69a0c97a854a78497d0c604732c24 with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash 19ed348f40b801a8127c5fc2c56c32b7 parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 789 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/6e9a0c9d-a0a2-4885-8a84-2a2022c961a0_calibrate_945868_52.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/6e9a0c9d-a0a2-4885-8a84-2a2022c961a0_calibrate_945868_52.stderr parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/99471206-f7fd-4b38-99d6-03f8c5cd105f_characterizeImage_945868_52.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/99471206-f7fd-4b38-99d6-03f8c5cd105f_characterizeImage_945868_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:28 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a144e8eeeff534603a064443a5e0fc8f with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash ef1bbf2d26c795e2e1aca6cfbbbd8894 parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 790 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/9ffff780-f814-4a67-9604-8d8c71468ace_calibrate_945755_19.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/9ffff780-f814-4a67-9604-8d8c71468ace_calibrate_945755_19.stderr parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/4c5bc8c7-0b72-472b-86d7-4130c9c5387b_characterizeImage_945755_19.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/4c5bc8c7-0b72-472b-86d7-4130c9c5387b_characterizeImage_945755_19.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07e4daf3f8d45bafc69d03541f00b336 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 8496381540c1d3d5a10a894ec06ce0d7 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 791 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/093a1fdd-5da7-4f3d-9a91-ad2118430097_calibrate_945869_56.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/093a1fdd-5da7-4f3d-9a91-ad2118430097_calibrate_945869_56.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/b3d46c05-8045-4775-a83e-f006ce86e7c2_characterizeImage_945869_56.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/b3d46c05-8045-4775-a83e-f006ce86e7c2_characterizeImage_945869_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:29 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2935b934b537d51201004b748ef9caf with result from task 745 parsl.dataflow.dflow DEBUG: Task 745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 746 has memoization hash 5e76f89fa85c1e1ae95e757ae5a37e85 parsl.dataflow.memoization INFO: Task 746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246312340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 792 parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8b886367-c0ff-48b5-a6b3-71d4747d2212_calibrate_945756_52.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8b886367-c0ff-48b5-a6b3-71d4747d2212_calibrate_945756_52.stderr parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/a8cc5b78-0b87-4953-8ee0-c80df3064dcd_characterizeImage_945756_52.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/a8cc5b78-0b87-4953-8ee0-c80df3064dcd_characterizeImage_945756_52.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b62cbdc9e9ed04f1e3d0778c07b57def with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 0ad06d7ccb8b8ba97259a2a6a07bbbcb parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 793 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/97bf7bb3-bc6b-442c-abf1-f6364953eab2_calibrate_945755_34.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/97bf7bb3-bc6b-442c-abf1-f6364953eab2_calibrate_945755_34.stderr parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/01809cc4-9ff1-4ce1-9493-b7ba4a91f98c_characterizeImage_945755_34.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/01809cc4-9ff1-4ce1-9493-b7ba4a91f98c_characterizeImage_945755_34.stderr parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1347c9ffc7c2b0435e29dcebc02ee70 with result from task 853 parsl.dataflow.dflow DEBUG: Task 853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 854 has memoization hash 3551c6671182b3b0b8b7034c8356d739 parsl.dataflow.memoization INFO: Task 854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 794 parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8729d606-33e5-4ba6-8547-aa3cdf6224b5_calibrate_945756_13.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8729d606-33e5-4ba6-8547-aa3cdf6224b5_calibrate_945756_13.stderr parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/066c3b35-648c-40ea-b979-ef2ba521535f_characterizeImage_945756_13.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/066c3b35-648c-40ea-b979-ef2ba521535f_characterizeImage_945756_13.stderr parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 877d0b8a3d75e81d2013d53f2448ec25 with result from task 820 parsl.dataflow.dflow DEBUG: Task 820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 821 has memoization hash 7b13657b411833cb6e79d4bf3e3e1518 parsl.dataflow.memoization INFO: Task 821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 795 parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/527221cc-7364-4bb8-afdd-920896b71071_calibrate_945756_10.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/527221cc-7364-4bb8-afdd-920896b71071_calibrate_945756_10.stderr parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ae7efdd8-b351-4590-a9c8-07879a426918_characterizeImage_945756_10.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ae7efdd8-b351-4590-a9c8-07879a426918_characterizeImage_945756_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:29 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6396d82ddfb35c0d279a4083a163ccf with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 43ef125382346a36ee9901e3a5a0a561 parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 796 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c7364b65-81fd-43c6-962b-33981f93ac01_calibrate_945756_53.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c7364b65-81fd-43c6-962b-33981f93ac01_calibrate_945756_53.stderr parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ba0808e8-c461-4f0d-8b4f-b22c688e534a_characterizeImage_945756_53.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/ba0808e8-c461-4f0d-8b4f-b22c688e534a_characterizeImage_945756_53.stderr parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e6ef66b78bd1e549e4b946fe422cfcd with result from task 595 parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash 11e1fd39ec42457748fdc4a33474ad19 parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246284d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 797 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/41094bfc-5685-4a86-a01a-94dbf9b6eb26_calibrate_945756_8.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/41094bfc-5685-4a86-a01a-94dbf9b6eb26_calibrate_945756_8.stderr parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6a945e68-db3b-4bbf-a28d-1b7c2a4aecd6_characterizeImage_945756_8.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6a945e68-db3b-4bbf-a28d-1b7c2a4aecd6_characterizeImage_945756_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:30 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 1093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5451cd1a46485afba39607e3fb84a742 with result from task 1093 parsl.dataflow.dflow DEBUG: Task 1093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1094 has memoization hash 834d0f4f464805abe05d5020d15ae514 parsl.dataflow.memoization INFO: Task 1094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1094 try 0 launched on executor multi with executor id 798 parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/f9cc263d-d182-49f6-8c8b-969a697fecf9_calibrate_945757_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/f9cc263d-d182-49f6-8c8b-969a697fecf9_calibrate_945757_4.stderr parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/2404b549-8be8-479e-8a78-e40964bc5e6c_characterizeImage_945757_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/2404b549-8be8-479e-8a78-e40964bc5e6c_characterizeImage_945757_4.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 383284de431821370c02ffdf2d90381a with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash 370e6fda0f044335c244c8fa6a6a7aa8 parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462be520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 799 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/c045ced5-96b3-4049-8da7-fb0795abbbed_calibrate_945868_40.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/c045ced5-96b3-4049-8da7-fb0795abbbed_calibrate_945868_40.stderr parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/764407ce-23da-460f-83c7-43d5e8ba9f5f_characterizeImage_945868_40.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/764407ce-23da-460f-83c7-43d5e8ba9f5f_characterizeImage_945868_40.stderr parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cd2fcad8352d0fcce6e25fb477457ba with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash 625b0f577c73c040039b5850cf30fc2a parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462851c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 800 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/2815e669-50aa-4517-91ff-95dc185287fb_calibrate_945757_3.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/2815e669-50aa-4517-91ff-95dc185287fb_calibrate_945757_3.stderr parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/c11d1a60-564e-4871-9281-667f61b6921b_characterizeImage_945757_3.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/c11d1a60-564e-4871-9281-667f61b6921b_characterizeImage_945757_3.stderr parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 529ff15e9de4e1451fa2be6c465c1b1e with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash bd94ce7f8e2aaeaf881fd87145479354 parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 801 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/08bdd090-b345-4da0-92af-3b5280ca716c_calibrate_945869_57.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/08bdd090-b345-4da0-92af-3b5280ca716c_calibrate_945869_57.stderr parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/8d025b50-a8ec-48a9-886f-5f55b1b528a9_characterizeImage_945869_57.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/8d025b50-a8ec-48a9-886f-5f55b1b528a9_characterizeImage_945869_57.stderr parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60d7a3809bdcac614c4a210af4a200cc with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash 119e6968764fe08f4355a1b77fee9330 parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 802 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/be984a3a-961e-4bad-9ebb-25f43abf0580_calibrate_945867_8.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/be984a3a-961e-4bad-9ebb-25f43abf0580_calibrate_945867_8.stderr parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4808126a-c807-41ab-8871-bc476ef84391_characterizeImage_945867_8.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4808126a-c807-41ab-8871-bc476ef84391_characterizeImage_945867_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:30 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2d84d82bd259f211e0931b9340fc7d2 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/2b3955b6-065b-44fc-b4f5-d51eddfab9db_calibrate_945757_55.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/2b3955b6-065b-44fc-b4f5-d51eddfab9db_calibrate_945757_55.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f70afbc4cc0e33b38e25698bc9590f8 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/21f0b060-9829-4aeb-9c89-bba7f45bc498_calibrate_945755_39.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/21f0b060-9829-4aeb-9c89-bba7f45bc498_calibrate_945755_39.stderr parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 482f9c6565adc8eafef413d7699b39e3 with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c52199d5-b7f0-40d8-bf16-0e06de5ec3d2_calibrate_945756_30.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c52199d5-b7f0-40d8-bf16-0e06de5ec3d2_calibrate_945756_30.stderr parsl.dataflow.dflow INFO: Task 1111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2e8c631106202c337f86074b84ca300 with result from task 1111 parsl.dataflow.dflow DEBUG: Task 1111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1112 has memoization hash fe950f75ea300015c1b397b5b6556b86 parsl.dataflow.memoization INFO: Task 1112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924610da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1112 try 0 launched on executor multi with executor id 803 parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d3f9e3b3-b2ee-4017-b90e-cc677d163c0d_calibrate_945755_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d3f9e3b3-b2ee-4017-b90e-cc677d163c0d_calibrate_945755_11.stderr parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/4673b9a6-9a17-462b-bab9-6ee730087fb4_characterizeImage_945755_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/4673b9a6-9a17-462b-bab9-6ee730087fb4_characterizeImage_945755_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:31 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94d46d08ffd08ac652947a65be9bfb33 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/643e846d-9b3d-460d-be18-ad59ed6df5d4_calibrate_945757_35.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/643e846d-9b3d-460d-be18-ad59ed6df5d4_calibrate_945757_35.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4132a3553dfa8164734d285d1f154c9a with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/4a7c5d2a-b089-49fe-a1f8-7e5afc96fa95_calibrate_945755_62.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/4a7c5d2a-b089-49fe-a1f8-7e5afc96fa95_calibrate_945755_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:32 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0627a026ed7c5bf155c73fdbb7133508 with result from task 823 parsl.dataflow.dflow DEBUG: Task 823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 824 has memoization hash 4a32ae9431a4c8ca046ed0ea58e7db81 parsl.dataflow.memoization INFO: Task 824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 804 parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/6290a584-d78c-4d3b-91a8-4750a30a2912_calibrate_945868_13.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/6290a584-d78c-4d3b-91a8-4750a30a2912_calibrate_945868_13.stderr parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/69f7205a-a3b6-4d1d-acf4-36fcf6d6a549_characterizeImage_945868_13.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/69f7205a-a3b6-4d1d-acf4-36fcf6d6a549_characterizeImage_945868_13.stderr parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83f2fe949445babcb01a601699a95073 with result from task 832 parsl.dataflow.dflow DEBUG: Task 832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 833 has memoization hash 59bf962114080461c6014ffd0307737d parsl.dataflow.memoization INFO: Task 833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 805 parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d61263c8-155b-49e7-a52a-ee1f4ce8aa55_calibrate_945757_18.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d61263c8-155b-49e7-a52a-ee1f4ce8aa55_calibrate_945757_18.stderr parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/814c29b8-cb7d-40dc-ab68-a71f77986dcf_characterizeImage_945757_18.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/814c29b8-cb7d-40dc-ab68-a71f77986dcf_characterizeImage_945757_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:32 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:33 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f90f5db473d2453d8f274e1ec86a499 with result from task 829 parsl.dataflow.dflow DEBUG: Task 829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 830 has memoization hash 4acb028ca933616cb75e2cf534002420 parsl.dataflow.memoization INFO: Task 830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 806 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/354e48d5-9ec8-4bb4-ab94-62b4f5f62c6c_calibrate_945869_37.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/354e48d5-9ec8-4bb4-ab94-62b4f5f62c6c_calibrate_945869_37.stderr parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/c9f57031-214e-41ec-bae6-053386243caf_characterizeImage_945869_37.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/c9f57031-214e-41ec-bae6-053386243caf_characterizeImage_945869_37.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b4f1cff0ec6ffd52c41d97bb0088dbc with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash c28db5f99bebd74bd52c6eb2c335d964 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247994e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 807 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/cd0eb8fb-ba62-42e8-9431-65f4100b9655_calibrate_945867_15.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/cd0eb8fb-ba62-42e8-9431-65f4100b9655_calibrate_945867_15.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/0da76e25-93cd-4345-8147-898b4f324eaa_characterizeImage_945867_15.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/0da76e25-93cd-4345-8147-898b4f324eaa_characterizeImage_945867_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:33 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afa8cfd6c33f17e7ce5412da390120cf with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 49d8beff6b60c54468bed653d3549507 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 808 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ee638eaf-408c-48a0-b011-d9224d65c4da_calibrate_945868_24.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ee638eaf-408c-48a0-b011-d9224d65c4da_calibrate_945868_24.stderr parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0e1aaf65-35df-4955-bd00-6394c384adbf_characterizeImage_945868_24.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0e1aaf65-35df-4955-bd00-6394c384adbf_characterizeImage_945868_24.stderr parsl.dataflow.dflow INFO: Task 1066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b42522db37b12170c07ca559e32e5af3 with result from task 1066 parsl.dataflow.dflow DEBUG: Task 1066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1067 has memoization hash 9382be0d0f2e4e9abb7c715687fd0cd7 parsl.dataflow.memoization INFO: Task 1067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1067 try 0 launched on executor multi with executor id 809 parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/263d0ba8-5c77-48b0-b234-cf6812eed249_calibrate_945757_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/263d0ba8-5c77-48b0-b234-cf6812eed249_calibrate_945757_52.stderr parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/7583a6f2-13cd-4250-b2a3-5c6ef5ae3105_characterizeImage_945757_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/7583a6f2-13cd-4250-b2a3-5c6ef5ae3105_characterizeImage_945757_52.stderr parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bda45173f8d6e9f7000f6aa24eac1768 with result from task 709 parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash 10e108d3730192717d1becb92943b071 parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246310180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 810 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/002c269d-5810-4e98-af3a-80ce8accb8e9_calibrate_945756_41.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/002c269d-5810-4e98-af3a-80ce8accb8e9_calibrate_945756_41.stderr parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/e4b3104e-fbc9-4eb2-8c1e-152e60e965f6_characterizeImage_945756_41.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/e4b3104e-fbc9-4eb2-8c1e-152e60e965f6_characterizeImage_945756_41.stderr parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f5a6708aa28b2554b295cf17a0e527f with result from task 826 parsl.dataflow.dflow DEBUG: Task 826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 827 has memoization hash 226339ac707ffd722b72063a9afadad8 parsl.dataflow.memoization INFO: Task 827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 811 parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2464f8e4-f2ec-4669-962e-d03f694b9849_calibrate_945868_46.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2464f8e4-f2ec-4669-962e-d03f694b9849_calibrate_945868_46.stderr parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a15981f6-22e2-41b1-9c41-dfc7a2f76fdb_characterizeImage_945868_46.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a15981f6-22e2-41b1-9c41-dfc7a2f76fdb_characterizeImage_945868_46.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd5cc0b386da2bb1bd4672333af248a0 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash fb3f4a22956121b1f3b0680b4139021c parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 812 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/787cac3f-a298-47f2-9b86-47dd451d529e_calibrate_945757_1.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/787cac3f-a298-47f2-9b86-47dd451d529e_calibrate_945757_1.stderr parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/0b584a4d-b5f9-4797-86c2-28ec77c4d646_characterizeImage_945757_1.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/0b584a4d-b5f9-4797-86c2-28ec77c4d646_characterizeImage_945757_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:34 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf8a8828c26257c2cb05f05624a7f25a with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d2b9f1f6-cc6e-4109-a3f1-991d49b09293_calibrate_945755_37.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d2b9f1f6-cc6e-4109-a3f1-991d49b09293_calibrate_945755_37.stderr parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42fb0f081e23a6395ab9cc382709199f with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash f1b026fc8f93220f0a08b67498b2da19 parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 813 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/544cb80c-b9ac-437a-a8c7-8aca24a23dad_calibrate_945757_11.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/544cb80c-b9ac-437a-a8c7-8aca24a23dad_calibrate_945757_11.stderr parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/3dd4ea47-fd9d-4e1c-b014-75061e41f46f_characterizeImage_945757_11.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/3dd4ea47-fd9d-4e1c-b014-75061e41f46f_characterizeImage_945757_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:34 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c25adc0c288e1860489bc59e7a380d97 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash af4406501d4c2a69dbd874b541b2ad83 parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 814 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/9df2b344-dde2-4632-82e2-571d3334835b_calibrate_945868_51.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/9df2b344-dde2-4632-82e2-571d3334835b_calibrate_945868_51.stderr parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/cfcb946f-09cb-41fd-81ac-b54e8f64e336_characterizeImage_945868_51.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/cfcb946f-09cb-41fd-81ac-b54e8f64e336_characterizeImage_945868_51.stderr parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2386eb0660300070584b8a0a01ab4dc with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/a839ef76-b482-453f-b06b-e1310a574441_calibrate_945755_35.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/a839ef76-b482-453f-b06b-e1310a574441_calibrate_945755_35.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7dfd4b0345f2819d07e08d60f314e88 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 2e2bfcae06ca753de6dbf38acf27581e parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247916b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 815 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f32fd2f5-f3b7-42f7-bd39-e0d5c0984ece_calibrate_945755_24.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f32fd2f5-f3b7-42f7-bd39-e0d5c0984ece_calibrate_945755_24.stderr parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/becaa19d-15b4-446f-a2c4-88ebf144089b_characterizeImage_945755_24.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/becaa19d-15b4-446f-a2c4-88ebf144089b_characterizeImage_945755_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:35 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:35 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:36 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c057a8ebafcb4b57509a61a11a40f46 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash a55eb2d19340a19030c56918fbe92287 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 816 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/108d6122-57ff-4fc3-bd11-96e1983822f0_calibrate_945868_60.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/108d6122-57ff-4fc3-bd11-96e1983822f0_calibrate_945868_60.stderr parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/caee9d8f-db79-48ca-9e09-98c915867ba1_characterizeImage_945868_60.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/caee9d8f-db79-48ca-9e09-98c915867ba1_characterizeImage_945868_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:36 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:37 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry daa8ca8755883e94bf8e797c7b040c99 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 6f3b88128843ff80d85b1582fa80cf15 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479976a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 817 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/6e82457c-4a42-4302-a077-1425c7357d5a_calibrate_945757_20.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/6e82457c-4a42-4302-a077-1425c7357d5a_calibrate_945757_20.stderr parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/2c414687-7bdf-4d59-9e50-ce61852c579d_characterizeImage_945757_20.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/2c414687-7bdf-4d59-9e50-ce61852c579d_characterizeImage_945757_20.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 712 try 1 failed parsl.dataflow.dflow ERROR: Task 712 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f576e43c2e0e9d807c6aa91822f77141 with result from task 712 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 713 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 713 try 0 failed parsl.dataflow.dflow INFO: Task 713 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/eace5a9d-9558-4834-be8f-9e8c0448f781_calibrate_945755_61.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/eace5a9d-9558-4834-be8f-9e8c0448f781_calibrate_945755_61.stderr parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/9b73461e-d2b8-49e7-8407-bbec1785b187_characterizeImage_945755_61.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/9b73461e-d2b8-49e7-8407-bbec1785b187_characterizeImage_945755_61.stderr parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 100f0b7c17cac6f2773b09747ff83dd0 with result from task 859 parsl.dataflow.dflow DEBUG: Task 859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 860 has memoization hash 113f1d838abcbf1b226503f3336d4e48 parsl.dataflow.memoization INFO: Task 860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aa160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 818 parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/20dab909-0299-468c-b217-6b1777232041_calibrate_945755_40.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/20dab909-0299-468c-b217-6b1777232041_calibrate_945755_40.stderr parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/6c0652fb-becd-4ca2-bd9c-59d4fca4e19c_characterizeImage_945755_40.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/6c0652fb-becd-4ca2-bd9c-59d4fca4e19c_characterizeImage_945755_40.stderr parsl.dataflow.dflow INFO: Task 970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 103f62c0a03ca52a1be27928f7801fc9 with result from task 970 parsl.dataflow.dflow DEBUG: Task 970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 971 has memoization hash f25823ce4ef29eef545f936b9df35f5e parsl.dataflow.memoization INFO: Task 971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 971 try 0 launched on executor multi with executor id 819 parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/e8bb4d2c-360f-447d-a721-2446f0838189_calibrate_945756_17.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/e8bb4d2c-360f-447d-a721-2446f0838189_calibrate_945756_17.stderr parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/eb5af35a-aa91-4585-ab54-68f4edfe7947_characterizeImage_945756_17.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/eb5af35a-aa91-4585-ab54-68f4edfe7947_characterizeImage_945756_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:38 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6a73372d4be802b240fa769993a5daf with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 21a2af663eaca4d11e1eb5b1f3fef868 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247996d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 820 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/ed7d12c3-bc48-42e3-8775-49ff0e398239_calibrate_945867_17.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/ed7d12c3-bc48-42e3-8775-49ff0e398239_calibrate_945867_17.stderr parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/5959f88e-dda5-40f8-aa21-b2bc8fb54de1_characterizeImage_945867_17.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/5959f88e-dda5-40f8-aa21-b2bc8fb54de1_characterizeImage_945867_17.stderr parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a86c11eaa2c77a502e435a2d777c8cdd with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 50d08e98d41dcce63097e316f9848499 parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bdbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 821 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/166489b5-7cb4-4b2c-b159-db335cbaf6c0_calibrate_945868_38.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/166489b5-7cb4-4b2c-b159-db335cbaf6c0_calibrate_945868_38.stderr parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/3e50b388-2dfa-4a98-aae7-01d790c4085a_characterizeImage_945868_38.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/3e50b388-2dfa-4a98-aae7-01d790c4085a_characterizeImage_945868_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:38 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:39 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02ddb1959c42d5d4a5e48dc22602a3a8 with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash 4000c658733fdc6633914a13b5dfbc62 parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246287380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 822 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f2723b48-d5fe-4264-b42d-27976316ae85_calibrate_945755_9.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f2723b48-d5fe-4264-b42d-27976316ae85_calibrate_945755_9.stderr parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/f8e7cad5-34f7-4672-9963-bc716c015bd7_characterizeImage_945755_9.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/f8e7cad5-34f7-4672-9963-bc716c015bd7_characterizeImage_945755_9.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a756e259f8bf4ab15e917b2320b090e9 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 50880a0290c04d1fea3bdf948e795b7f parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247995a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:39 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 823 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/878ab9fd-9483-4bd7-88ff-ee935f59573b_calibrate_945867_42.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/878ab9fd-9483-4bd7-88ff-ee935f59573b_calibrate_945867_42.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e2e8556a-8ead-4014-8072-023d806b35d0_characterizeImage_945867_42.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e2e8556a-8ead-4014-8072-023d806b35d0_characterizeImage_945867_42.stderr parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8480db1fe62799f452e2fadee32ff4f with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash b2067f270835d5d77855c5dae4740650 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479159e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 824 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/ac012908-d11a-4e4a-bd64-1751627fa4c1_calibrate_945757_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/ac012908-d11a-4e4a-bd64-1751627fa4c1_calibrate_945757_54.stderr parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/13b3bed6-f3ab-42f7-9301-0377f9c0090d_characterizeImage_945757_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/13b3bed6-f3ab-42f7-9301-0377f9c0090d_characterizeImage_945757_54.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be28ee4d063cf481ed09994a64552bea with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash de8cffe558c7ba67a3bb4174c50df439 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 825 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/99a69e7f-ba6b-4fad-8996-427c269854ab_calibrate_945757_43.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/99a69e7f-ba6b-4fad-8996-427c269854ab_calibrate_945757_43.stderr parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dd0270c2-5486-4a9a-949b-c42465a79800_characterizeImage_945757_43.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dd0270c2-5486-4a9a-949b-c42465a79800_characterizeImage_945757_43.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9665dbb01316a4920d1a9b41ddccdd8f with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4dec484a-5e5c-47e8-88b9-0be890298a82_calibrate_945757_40.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4dec484a-5e5c-47e8-88b9-0be890298a82_calibrate_945757_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:40 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ae76d038e081481712d88c791530f42 with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/8803601b-9101-4b2e-9a28-9409d086c601_calibrate_945755_21.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/8803601b-9101-4b2e-9a28-9409d086c601_calibrate_945755_21.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 315182a2d8da5f6a2ae0ad6385aacb3e with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/c97f2829-28eb-4ee4-a30a-e67f50c4ee99_calibrate_945757_32.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/c97f2829-28eb-4ee4-a30a-e67f50c4ee99_calibrate_945757_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:40 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77d4371925aa6070414f30913ab32d1e with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/e95f3031-c1c6-4643-96a6-27e914816fc3_calibrate_945757_56.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/e95f3031-c1c6-4643-96a6-27e914816fc3_calibrate_945757_56.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 336da3fc0001dee0724aed037b4a3591 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 3ff8a12b791a551835e95aec37b8ad4d parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 826 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/da0bd654-daaa-4923-aab5-c9fc713c42a2_calibrate_945757_45.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/da0bd654-daaa-4923-aab5-c9fc713c42a2_calibrate_945757_45.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/c86ca45e-8a06-4332-a812-96dc36252dda_characterizeImage_945757_45.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/c86ca45e-8a06-4332-a812-96dc36252dda_characterizeImage_945757_45.stderr parsl.dataflow.dflow INFO: Task 940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d13ac42f0cda595186584a92e4958b2 with result from task 940 parsl.dataflow.dflow DEBUG: Task 940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 941 has memoization hash d4791abebe38f99b596afa547d67fe97 parsl.dataflow.memoization INFO: Task 941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462287c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 941 try 0 launched on executor multi with executor id 827 parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/a3419303-94c3-4f56-82dd-6f156f2c4d2b_calibrate_945757_58.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/a3419303-94c3-4f56-82dd-6f156f2c4d2b_calibrate_945757_58.stderr parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/38a89c5d-a79c-4dd8-8666-163d051adf76_characterizeImage_945757_58.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/38a89c5d-a79c-4dd8-8666-163d051adf76_characterizeImage_945757_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:41 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b95fbb279f3d9b9ac556e081f0fbc7d5 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 89d68c276cb2cde9c69996f37455d7ca parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247995da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 828 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/685381e8-6baa-418b-b28e-d86c9ef40c49_calibrate_945867_56.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/685381e8-6baa-418b-b28e-d86c9ef40c49_calibrate_945867_56.stderr parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/32abffb1-a804-4322-9f63-616a8b4f5dda_characterizeImage_945867_56.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/32abffb1-a804-4322-9f63-616a8b4f5dda_characterizeImage_945867_56.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 415 try 1 failed parsl.dataflow.dflow ERROR: Task 415 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 08e0326b6388cd3bc6e195eb31b18119 with result from task 415 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 416 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 416 try 0 failed parsl.dataflow.dflow INFO: Task 416 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/1874c87e-54aa-4549-8f4c-3556f3d4a9a8_calibrate_945757_61.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/1874c87e-54aa-4549-8f4c-3556f3d4a9a8_calibrate_945757_61.stderr parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/8d904e3f-87f0-4602-b5f5-7c9327a328bf_characterizeImage_945757_61.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/8d904e3f-87f0-4602-b5f5-7c9327a328bf_characterizeImage_945757_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:41 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a4fe2956a8216ac5a21cd390dea4955 with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/5098e02b-7b60-4ee9-8ff4-36d68f341163_calibrate_945757_46.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/5098e02b-7b60-4ee9-8ff4-36d68f341163_calibrate_945757_46.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41b08934c51f77bb295dc4d2107b002e with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 30ab1ac82e4eb6133c49589fbb9a7b70 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 829 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/afc2a44f-cb76-4dca-b642-185ea174ea84_calibrate_945868_49.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/afc2a44f-cb76-4dca-b642-185ea174ea84_calibrate_945868_49.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/c77b24e4-6910-4773-831a-208a9ef6e713_characterizeImage_945868_49.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/c77b24e4-6910-4773-831a-208a9ef6e713_characterizeImage_945868_49.stderr parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62e0795290e460b0c7f78d63e701e40f with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/36409f61-e809-49d0-92b3-3d61f7b39787_calibrate_945757_6.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/36409f61-e809-49d0-92b3-3d61f7b39787_calibrate_945757_6.stderr parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12094b391d6ce2d1293d6e016c45d90f with result from task 778 parsl.dataflow.dflow DEBUG: Task 778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 779 has memoization hash acd3ccfc783d1b77f5e2516edee24e4c parsl.dataflow.memoization INFO: Task 779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 830 parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/064e3f3e-12f1-454c-a18b-f82e90c6bfcb_calibrate_945755_43.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/064e3f3e-12f1-454c-a18b-f82e90c6bfcb_calibrate_945755_43.stderr parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/b8d5b54d-db75-4808-9cb4-479a2f80056e_characterizeImage_945755_43.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/b8d5b54d-db75-4808-9cb4-479a2f80056e_characterizeImage_945755_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:42 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7519a1f4246e4b66777f23449a7d297c with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash c8b14fc67e82fa958f4a357535109438 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 831 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/140b045d-d01d-4c40-ba4d-945d886754be_calibrate_945757_48.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/140b045d-d01d-4c40-ba4d-945d886754be_calibrate_945757_48.stderr parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/36eef5ea-60d7-4608-9ad4-2a96254bbbfc_characterizeImage_945757_48.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/36eef5ea-60d7-4608-9ad4-2a96254bbbfc_characterizeImage_945757_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:43 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86b5c07678188607a4e31d06c61e769f with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 561e90360b35170a45296d32029d8a2d parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 832 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/5c326abd-23a8-4a24-ab94-726eef669a92_calibrate_945869_24.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/5c326abd-23a8-4a24-ab94-726eef669a92_calibrate_945869_24.stderr parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/6876be82-4175-426f-bca0-ccb3b08f9f12_characterizeImage_945869_24.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/6876be82-4175-426f-bca0-ccb3b08f9f12_characterizeImage_945869_24.stderr parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 384f3414f38d26306685e305e720dae4 with result from task 875 parsl.dataflow.dflow DEBUG: Task 875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/48f21c4a-2052-49e9-8ce3-08f3938d6fe2_calibrate_945755_8.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/48f21c4a-2052-49e9-8ce3-08f3938d6fe2_calibrate_945755_8.stderr parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9d61b774f15de3d111ccd294c7f1d52 with result from task 769 parsl.dataflow.dflow DEBUG: Task 769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 770 has memoization hash d3235f85325d58cfd9e574092b0e2891 parsl.dataflow.memoization INFO: Task 770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246313920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 833 parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/a7d1b8d9-c6fb-41ab-bd92-9539c1e61b00_calibrate_945756_42.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/a7d1b8d9-c6fb-41ab-bd92-9539c1e61b00_calibrate_945756_42.stderr parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/240e9e59-915c-4f6e-bef2-1605cd20f2ba_characterizeImage_945756_42.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/240e9e59-915c-4f6e-bef2-1605cd20f2ba_characterizeImage_945756_42.stderr parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6144d3f6199cf5e9df9828953672240 with result from task 775 parsl.dataflow.dflow DEBUG: Task 775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 776 has memoization hash bc444c226ef44591feab997f085b2b06 parsl.dataflow.memoization INFO: Task 776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 834 parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c3e46a8d-74cd-4e4c-9948-4a9ac6332350_calibrate_945756_21.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c3e46a8d-74cd-4e4c-9948-4a9ac6332350_calibrate_945756_21.stderr parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f9cbb8a9-2503-467f-b14f-5742d8c0c764_characterizeImage_945756_21.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f9cbb8a9-2503-467f-b14f-5742d8c0c764_characterizeImage_945756_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:43 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e17887fb5d6f08cc166e971d1cc7359 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 38fb5e5fe5a0cddbfd223bde616d4432 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 835 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/0bb6ba42-6c3b-4d3e-a112-04046b346a54_calibrate_945756_62.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/0bb6ba42-6c3b-4d3e-a112-04046b346a54_calibrate_945756_62.stderr parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f472cb4b-8811-4352-a411-3ba2e52087b9_characterizeImage_945756_62.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f472cb4b-8811-4352-a411-3ba2e52087b9_characterizeImage_945756_62.stderr parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93afe3441f2b26968de04b53ceaaaffe with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/f53899a3-33db-46f3-8402-aa9283292a43_calibrate_945757_41.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/f53899a3-33db-46f3-8402-aa9283292a43_calibrate_945757_41.stderr parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00620e11b673f181d001d9366cc2b98e with result from task 754 parsl.dataflow.dflow DEBUG: Task 754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 755 has memoization hash 052fb6949a727dfb3112fd7ee0df082f parsl.dataflow.memoization INFO: Task 755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246312c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 836 parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/6c090abd-d51d-4aae-8d5f-1135456a8f6a_calibrate_945756_37.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/6c090abd-d51d-4aae-8d5f-1135456a8f6a_calibrate_945756_37.stderr parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/e167eab4-fd24-403e-8180-30daf7ae4bb9_characterizeImage_945756_37.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/e167eab4-fd24-403e-8180-30daf7ae4bb9_characterizeImage_945756_37.stderr parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0a1bc030e2f9d2e644b785ac780169e with result from task 901 parsl.dataflow.dflow DEBUG: Task 901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 347d34457680375a95e340c27e683474 parsl.dataflow.memoization INFO: Task 902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 837 parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/9eb9759a-66e0-482a-8f28-4127e880b901_calibrate_945757_10.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/9eb9759a-66e0-482a-8f28-4127e880b901_calibrate_945757_10.stderr parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/7617e05d-5638-460a-96bf-b33bf1b2851a_characterizeImage_945757_10.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/7617e05d-5638-460a-96bf-b33bf1b2851a_characterizeImage_945757_10.stderr parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d37b347ba0d34e9722a61e86b921f8a with result from task 565 parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:44 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash d89f7aba6f2d70d4690bb86d966bca6e parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 838 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b4913521-fdd4-4b6f-aacf-90c5fa2e6df6_calibrate_945869_55.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b4913521-fdd4-4b6f-aacf-90c5fa2e6df6_calibrate_945869_55.stderr parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/714f1228-0972-4bd0-b39b-8581c6b8d9a0_characterizeImage_945869_55.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/714f1228-0972-4bd0-b39b-8581c6b8d9a0_characterizeImage_945869_55.stderr parsl.dataflow.dflow INFO: Task 1058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e9ef6a9f743d7ffdafd51ce5de39a30 with result from task 1058 parsl.dataflow.dflow DEBUG: Task 1058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/46048db4-e96d-4cfc-a540-7d5dae959db6_calibrate_945755_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/46048db4-e96d-4cfc-a540-7d5dae959db6_calibrate_945755_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:44 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec5af748ad530dd4dce3bbc13ee92b7f with result from task 878 parsl.dataflow.dflow DEBUG: Task 878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/932567e1-e46f-4ad1-8b49-98c6a0d843c9_calibrate_945755_28.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/932567e1-e46f-4ad1-8b49-98c6a0d843c9_calibrate_945755_28.stderr parsl.dataflow.dflow INFO: Task 1099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0b1603535d57500270d224be3be964f with result from task 1099 parsl.dataflow.dflow DEBUG: Task 1099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1100 has memoization hash fdb9a36c7e372ca4c80e279d569d1d22 parsl.dataflow.memoization INFO: Task 1100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1100 try 0 launched on executor multi with executor id 839 parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/b37118f2-5e9c-47eb-901c-2e2fdfedb6c4_calibrate_945757_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/b37118f2-5e9c-47eb-901c-2e2fdfedb6c4_calibrate_945757_53.stderr parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b79a326e-66a4-4ab8-a43e-e568e6ba0fef_characterizeImage_945757_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/b79a326e-66a4-4ab8-a43e-e568e6ba0fef_characterizeImage_945757_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:45 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c253a36d347688fc8282e2d3831e931 with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/ca9ee91b-a171-4669-8b31-11e719648560_calibrate_945755_5.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/ca9ee91b-a171-4669-8b31-11e719648560_calibrate_945755_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:45 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5541fad15dc5d169a3d3c34bceb5d1bd with result from task 943 parsl.dataflow.dflow DEBUG: Task 943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 944 has memoization hash 78d3076c9d2f008831be903ab9d76f23 parsl.dataflow.memoization INFO: Task 944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246228400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 944 try 0 launched on executor multi with executor id 840 parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/1888263a-48de-4176-ba61-a7a7bc025ab2_calibrate_945756_51.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/1888263a-48de-4176-ba61-a7a7bc025ab2_calibrate_945756_51.stderr parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/d2dcceb0-ef83-41ab-b355-340164ee6ac9_characterizeImage_945756_51.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/d2dcceb0-ef83-41ab-b355-340164ee6ac9_characterizeImage_945756_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:46 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea68f103b89fca8a0fd26cbb6c15f67d with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash eaedaaaac09ed666da62a9668d5c24d1 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 841 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/db7480ef-3d40-4420-bf8e-6c2612e2653f_calibrate_945757_7.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/db7480ef-3d40-4420-bf8e-6c2612e2653f_calibrate_945757_7.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dedf45af-d869-4edf-9965-3981ddb78f00_characterizeImage_945757_7.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dedf45af-d869-4edf-9965-3981ddb78f00_characterizeImage_945757_7.stderr parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e15c9d434a5c65cfb7f631d9f9d4f334 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/514125a3-435d-49f3-9eac-5c875a809add_calibrate_945756_5.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/514125a3-435d-49f3-9eac-5c875a809add_calibrate_945756_5.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd798ef359f9c6269a819a817f045d1d with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/a6a8d124-98b7-4bac-b266-7818af939257_calibrate_945757_50.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/a6a8d124-98b7-4bac-b266-7818af939257_calibrate_945757_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:47 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0bdc9d66a915e4feb119a12b5ba68b4 with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash 8f5e978a36c5a3821f83799bb58ebb04 parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 842 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0942ee44-be77-4467-8eea-cf0b171642f7_calibrate_945869_30.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0942ee44-be77-4467-8eea-cf0b171642f7_calibrate_945869_30.stderr parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2b746d7f-f3ad-4d82-a086-b1cbaeda084b_characterizeImage_945869_30.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2b746d7f-f3ad-4d82-a086-b1cbaeda084b_characterizeImage_945869_30.stderr parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82eae41a444f223349934896396df377 with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/b9343450-1db9-4316-91a5-e23053d13084_calibrate_945755_4.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/b9343450-1db9-4316-91a5-e23053d13084_calibrate_945755_4.stderr parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93b9fd65d4a2cd441d921b96b0c5c5ed with result from task 904 parsl.dataflow.dflow DEBUG: Task 904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 905 has memoization hash a1491f8bb21d6fb5a154522303538f36 parsl.dataflow.memoization INFO: Task 905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 843 parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/e294ed5b-a034-408e-b6e5-b5e46f6851ee_calibrate_945757_12.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/e294ed5b-a034-408e-b6e5-b5e46f6851ee_calibrate_945757_12.stderr parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a77e7a17-10bc-4a16-96a6-0d1c55956e88_characterizeImage_945757_12.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a77e7a17-10bc-4a16-96a6-0d1c55956e88_characterizeImage_945757_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:47 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76d17711eecb491969034d285c489294 with result from task 838 parsl.dataflow.dflow DEBUG: Task 838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 839 has memoization hash 796ad19ee600a826597a864184b90a5e parsl.dataflow.memoization INFO: Task 839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 844 parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/a96279ca-6e03-4bf7-9786-1cc25eb5bc7f_calibrate_945868_48.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/a96279ca-6e03-4bf7-9786-1cc25eb5bc7f_calibrate_945868_48.stderr parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/c9b20239-a926-41d1-bf32-582de2e5334b_characterizeImage_945868_48.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/c9b20239-a926-41d1-bf32-582de2e5334b_characterizeImage_945868_48.stderr parsl.dataflow.dflow INFO: Task 1091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02c4767238506fda9210a4bb25b81f34 with result from task 1091 parsl.dataflow.dflow DEBUG: Task 1091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/189d78e6-3e29-44ce-8efa-f3d9f2f21881_calibrate_945755_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/189d78e6-3e29-44ce-8efa-f3d9f2f21881_calibrate_945755_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:48 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42afbb2239ab8ea356575c58fe9249bc with result from task 761 parsl.dataflow.dflow DEBUG: Task 761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/92744ff0-cef6-48c6-9b25-ba901fcdd950_calibrate_945756_59.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/92744ff0-cef6-48c6-9b25-ba901fcdd950_calibrate_945756_59.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fed1c0b25d94501aac9af387d6a8983 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/0de86e53-5f97-4887-a241-ac4db6aef5ed_calibrate_945756_54.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/0de86e53-5f97-4887-a241-ac4db6aef5ed_calibrate_945756_54.stderr parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e7f00ba145af188c74d4396e4f41e0b with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/55433490-2383-4ca0-ad25-044b8bef9082_calibrate_945755_45.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/55433490-2383-4ca0-ad25-044b8bef9082_calibrate_945755_45.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e905210b0f01681c852aca890dac166 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 6cb1de873bd90fc68d0a649871dbacf6 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 845 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f57be46e-99b0-4c7e-9691-60bc51265fcd_calibrate_945867_45.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f57be46e-99b0-4c7e-9691-60bc51265fcd_calibrate_945867_45.stderr parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/8e0faa55-d9fe-4f32-bbb5-3ce52383b816_characterizeImage_945867_45.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/8e0faa55-d9fe-4f32-bbb5-3ce52383b816_characterizeImage_945867_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:48 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 343c3b374000ea6e0bef68e17cd21686 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/b6be87e9-89a7-42c9-9db3-6cdfb5eb2095_calibrate_945755_59.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/b6be87e9-89a7-42c9-9db3-6cdfb5eb2095_calibrate_945755_59.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87c296e038dd895f7ec76f88b9824ef3 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/fbbe104d-a8fb-4221-8e72-d113e81caf81_calibrate_945755_23.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/fbbe104d-a8fb-4221-8e72-d113e81caf81_calibrate_945755_23.stderr parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81ab2da9fb3a3495ae5212b6265bd602 with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 9f992fd89410464843402dd437ba4895 parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462879c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 846 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/8e7cb3f7-9a3e-46c5-a180-7fde06f5dc98_calibrate_945755_60.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/8e7cb3f7-9a3e-46c5-a180-7fde06f5dc98_calibrate_945755_60.stderr parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/935de672-7f32-44c4-b943-17e3a5f92d8f_characterizeImage_945755_60.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/935de672-7f32-44c4-b943-17e3a5f92d8f_characterizeImage_945755_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:49 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:49 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccf031f3720148fd6cf1d49a78d7ee1b with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/d08eded8-f669-4e04-832c-49c01d3375a5_calibrate_945756_57.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/d08eded8-f669-4e04-832c-49c01d3375a5_calibrate_945756_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:50 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:50 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc5cecd431e6bab42c50f7093c3fbcc3 with result from task 908 parsl.dataflow.dflow DEBUG: Task 908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/08ad4f1f-611d-4433-8070-2b7f8b6cdd10_calibrate_945755_46.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/08ad4f1f-611d-4433-8070-2b7f8b6cdd10_calibrate_945755_46.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3dfce8eb6e57a3d399368e6d49480ff with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/45a470f3-0ffb-4a86-b6b4-84c8f4853bc3_calibrate_945757_49.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/45a470f3-0ffb-4a86-b6b4-84c8f4853bc3_calibrate_945757_49.stderr parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 534d7ec3c0fe5b8439a362fffdc49488 with result from task 799 parsl.dataflow.dflow DEBUG: Task 799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash 79308fbab5ec65825fc39130fd3ca90a parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463462a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 847 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/678dd4aa-5b29-4461-a57c-4dd9c36f8086_calibrate_945755_17.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/678dd4aa-5b29-4461-a57c-4dd9c36f8086_calibrate_945755_17.stderr parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/19e5fd0a-6567-45c2-89de-6b4872222f75_characterizeImage_945755_17.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/19e5fd0a-6567-45c2-89de-6b4872222f75_characterizeImage_945755_17.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 157 try 1 failed parsl.dataflow.dflow ERROR: Task 157 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 45db7ac2a82f88bd80ff37bac26a2d4a with result from task 157 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 158 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 158 try 0 failed parsl.dataflow.dflow INFO: Task 158 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/74006080-9770-4c90-8a06-911f7ab50cc6_calibrate_945756_61.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/74006080-9770-4c90-8a06-911f7ab50cc6_calibrate_945756_61.stderr parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/109edcee-de71-4665-a0bb-9c8ade9f452d_characterizeImage_945756_61.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/109edcee-de71-4665-a0bb-9c8ade9f452d_characterizeImage_945756_61.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23fac2d5b15739bc31698ceb4772c530 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 5bc5458ad97a542649ec8f2619e108d4 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 848 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/985613c8-69a4-4c9e-9844-6661b5a1f0c3_calibrate_945868_19.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/985613c8-69a4-4c9e-9844-6661b5a1f0c3_calibrate_945868_19.stderr parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/9d062a4d-cdd5-40f4-ac46-61304a70fef8_characterizeImage_945868_19.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/9d062a4d-cdd5-40f4-ac46-61304a70fef8_characterizeImage_945868_19.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d539fb9042047df36760a9111053fce4 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/f99242b3-5713-4352-9e53-dcad61ba6ad0_calibrate_945757_17.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/f99242b3-5713-4352-9e53-dcad61ba6ad0_calibrate_945757_17.stderr parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d2b6da22a5b50deb11e1cec454ea9d4 with result from task 611 parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/ddf97f43-5f7a-4806-a5a5-556ea0cf5b96_calibrate_945755_22.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/ddf97f43-5f7a-4806-a5a5-556ea0cf5b96_calibrate_945755_22.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa14f9d638a47bdb84299018a8a7380c with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7ab84536-01bd-40be-874f-05012a579d5c_calibrate_945755_20.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7ab84536-01bd-40be-874f-05012a579d5c_calibrate_945755_20.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 295f7143b0314c8f0a86800a60bbd9db with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/4f8cee6f-3055-41cf-a752-4aa2358952a4_calibrate_945755_7.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/4f8cee6f-3055-41cf-a752-4aa2358952a4_calibrate_945755_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:51 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c32d831971062837dd8b1703e99ed644 with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash cd8ea1a6f4b2ed65bced04c83ec6d4a4 parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bfb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 849 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/38174a41-78b0-4844-b9ae-68504bfadf37_calibrate_945868_34.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/38174a41-78b0-4844-b9ae-68504bfadf37_calibrate_945868_34.stderr parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/4869a680-bf71-4cb6-b385-138d029eac3f_characterizeImage_945868_34.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/4869a680-bf71-4cb6-b385-138d029eac3f_characterizeImage_945868_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:51 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf38ca7cf1823a54a813e0144b4bce18 with result from task 1103 parsl.dataflow.dflow DEBUG: Task 1103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/fd6f9235-b2b0-4510-9913-822b83a09c27_calibrate_945756_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/fd6f9235-b2b0-4510-9913-822b83a09c27_calibrate_945756_28.stderr parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e93c8127ede954d5d2fcdbcc8733938 with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/010cf681-91da-468d-9dc0-069aaeb81d25_calibrate_945757_38.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/010cf681-91da-468d-9dc0-069aaeb81d25_calibrate_945757_38.stderr parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7019a67d51883d54a92076151805609 with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash dbe68efac3d066fac9c80ae36bf3e243 parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 850 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/b672ddc8-3857-47d1-953f-803a34aacce7_calibrate_945867_1.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/b672ddc8-3857-47d1-953f-803a34aacce7_calibrate_945867_1.stderr parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/df86df58-d380-460d-820f-dca9138852a1_characterizeImage_945867_1.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/df86df58-d380-460d-820f-dca9138852a1_characterizeImage_945867_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:52 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dd12c154a48ba79abfae90c3f428ddd with result from task 1055 parsl.dataflow.dflow DEBUG: Task 1055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/b3d2224b-65cc-48ff-a763-bedbff3f3600_calibrate_945755_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/b3d2224b-65cc-48ff-a763-bedbff3f3600_calibrate_945755_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:52 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d50a6c909f12b6e2fb8e55f6c402d341 with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/91870a96-6f47-4f2d-9918-f93bd76a5544_calibrate_945755_57.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/91870a96-6f47-4f2d-9918-f93bd76a5544_calibrate_945755_57.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe3aedaa1d6d6f9e1487a518c541ddfc with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/ad7a8f04-557f-4b7a-8195-c893f2de0e09_calibrate_945756_35.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/ad7a8f04-557f-4b7a-8195-c893f2de0e09_calibrate_945756_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:53 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a14e4610488697fb9d470f35439214c6 with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash b74b9ebd657a173316226ebe68b8d290 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 851 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/bbd84d5a-9805-431d-b28b-75774f91659a_calibrate_945756_9.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/bbd84d5a-9805-431d-b28b-75774f91659a_calibrate_945756_9.stderr parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/0e076001-03f3-44f8-a9ea-968a3299c4ca_characterizeImage_945756_9.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/0e076001-03f3-44f8-a9ea-968a3299c4ca_characterizeImage_945756_9.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 914cb9f1d086afb9ca0909116fc346d3 with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 07e5b1eb7eb840d92210a9db5dbf5c31 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246392c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 852 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d1e71f44-6085-4ff6-ba0d-e99a73eb93a6_calibrate_945755_3.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d1e71f44-6085-4ff6-ba0d-e99a73eb93a6_calibrate_945755_3.stderr parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/07c2f948-bec9-4249-a2b4-1a087e4540d8_characterizeImage_945755_3.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/07c2f948-bec9-4249-a2b4-1a087e4540d8_characterizeImage_945755_3.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1502e909e47a4284e023602247c3639c with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 79968c11e7252038929392ce01f192c3 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246391300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 853 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1a02f2d3-77c1-4399-a941-aa05d4cca921_calibrate_945869_26.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1a02f2d3-77c1-4399-a941-aa05d4cca921_calibrate_945869_26.stderr parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1de467c6-2bd2-4ad1-97f3-a11a138c10eb_characterizeImage_945869_26.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1de467c6-2bd2-4ad1-97f3-a11a138c10eb_characterizeImage_945869_26.stderr parsl.dataflow.dflow INFO: Task 1088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3ff1e5e7571de64c3abe1eeeb2e9b54 with result from task 1088 parsl.dataflow.dflow DEBUG: Task 1088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/4df201b2-cad8-4d32-8402-1826f1091a56_calibrate_945756_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/4df201b2-cad8-4d32-8402-1826f1091a56_calibrate_945756_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:53 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c20fad654cb8cecd590d93e4df79f50 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/6e13b681-ed1c-4a2f-bb1d-4918ed335844_calibrate_945755_15.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/6e13b681-ed1c-4a2f-bb1d-4918ed335844_calibrate_945755_15.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef1bbf2d26c795e2e1aca6cfbbbd8894 with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/9ffff780-f814-4a67-9604-8d8c71468ace_calibrate_945755_19.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/9ffff780-f814-4a67-9604-8d8c71468ace_calibrate_945755_19.stderr parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8ed1f3c310ffb1d4a27b270beca359e with result from task 844 parsl.dataflow.dflow DEBUG: Task 844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 845 has memoization hash cca47ea38f34b056e24a7eb412051b33 parsl.dataflow.memoization INFO: Task 845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 854 parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/da524889-dab4-487e-855c-215be3efd9aa_calibrate_945756_49.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/da524889-dab4-487e-855c-215be3efd9aa_calibrate_945756_49.stderr parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/7972cfd9-c795-46bc-8414-bde02cceefa3_characterizeImage_945756_49.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/7972cfd9-c795-46bc-8414-bde02cceefa3_characterizeImage_945756_49.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1dd4e932acc5267908919513166f0d07 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 0032340e69cdd44ae76fe9194141f33e parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 855 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/ca39a6b5-520b-4d44-9ee2-82af7190bc26_calibrate_945869_20.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/ca39a6b5-520b-4d44-9ee2-82af7190bc26_calibrate_945869_20.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/49893688-6cc2-43a9-953a-40a926fb68b7_characterizeImage_945869_20.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/49893688-6cc2-43a9-953a-40a926fb68b7_characterizeImage_945869_20.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc5b24cadef89889098480cd0a605717 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash fb87fd6c4c958e06ad7169dcc7a962ea parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463922a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 856 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/166854bf-b992-4661-bc84-d27716f85bd7_calibrate_945868_12.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/166854bf-b992-4661-bc84-d27716f85bd7_calibrate_945868_12.stderr parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/ef599eda-3498-4034-bc62-c052e9bdc925_characterizeImage_945868_12.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/ef599eda-3498-4034-bc62-c052e9bdc925_characterizeImage_945868_12.stderr parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f299d525c84f98e4d4c666bc2f6d6750 with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/c6f3f557-e01d-4e30-a391-7bc480aaa7ad_calibrate_945755_26.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/c6f3f557-e01d-4e30-a391-7bc480aaa7ad_calibrate_945755_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:54 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5cfe13743c89a427ff25810cd6084d3 with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 79b747af09acee318c5b44337b86297a parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 857 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/18a41296-398c-449e-b7a7-d00591d488fd_characterizeImage_945757_29.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/18a41296-398c-449e-b7a7-d00591d488fd_characterizeImage_945757_29.stderr parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/ac0e44e0-f40e-4aaa-811e-bc1f9c48f561_isr_945757_29.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/ac0e44e0-f40e-4aaa-811e-bc1f9c48f561_isr_945757_29.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac92d6369ea0308e726ef66c3752fd15 with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash fb23b14992a0431f6fe77ce967f3e4fc parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246400540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 858 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/5e488abd-6d73-4d04-b66a-3bc712c9f5ac_characterizeImage_945869_35.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/5e488abd-6d73-4d04-b66a-3bc712c9f5ac_characterizeImage_945869_35.stderr parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/34c96422-0d9f-405c-a88d-f240882c3ff2_isr_945869_35.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/34c96422-0d9f-405c-a88d-f240882c3ff2_isr_945869_35.stderr parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 642667704e2854ac92066ecc0411fbe8 with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash fdae1e5cbfac39c13c3bc4afce8415e2 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246400ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 859 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a6138588-6fff-449f-9079-639b0c79a6cb_characterizeImage_945868_33.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a6138588-6fff-449f-9079-639b0c79a6cb_characterizeImage_945868_33.stderr parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/2f32c7f2-41a5-49c4-bb89-fcc701e874ca_isr_945868_33.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/2f32c7f2-41a5-49c4-bb89-fcc701e874ca_isr_945868_33.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32c3ffe342c35f6e1c52a25acdc6af5b with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 48fc672a5f1c88044e2476b4259ed183 parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 860 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/247d964f-0c0c-43b4-b9e2-43a8068c5b2f_characterizeImage_945867_43.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/247d964f-0c0c-43b4-b9e2-43a8068c5b2f_characterizeImage_945867_43.stderr parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/bec9c9ab-1d70-4190-aaf6-89807b11962a_isr_945867_43.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/bec9c9ab-1d70-4190-aaf6-89807b11962a_isr_945867_43.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bca3d7be065a9c173d6536a4ae7b34d with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash 993f308a478d4ddee21bc50462fa98e7 parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 861 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/fd14c9c4-9c6b-4b39-bf76-6f11d49dec5f_characterizeImage_945756_24.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/fd14c9c4-9c6b-4b39-bf76-6f11d49dec5f_characterizeImage_945756_24.stderr parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/bc0caf00-b0c9-4076-bd44-0b297772fefc_isr_945756_24.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/bc0caf00-b0c9-4076-bd44-0b297772fefc_isr_945756_24.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f47af81d18ad704e95902ebb31b3c907 with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash dd176b14c30e1fb08f3a4c31f9afd614 parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246400860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 862 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/3ef85d6b-ec2b-4c6b-bdc3-b2ac816e57ed_characterizeImage_945868_1.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/3ef85d6b-ec2b-4c6b-bdc3-b2ac816e57ed_characterizeImage_945868_1.stderr parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/44c25584-b173-452e-9cdd-96eb793aa963_isr_945868_1.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/44c25584-b173-452e-9cdd-96eb793aa963_isr_945868_1.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8472db3f4d13d5dd03ae4a4cdc2e4e83 with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 8ea559879396985f63a4664612659b00 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246400b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 863 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/305d1d01-e448-4b9c-aab5-d7490be37fe1_characterizeImage_945869_52.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/305d1d01-e448-4b9c-aab5-d7490be37fe1_characterizeImage_945869_52.stderr parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/dfd89499-cf54-48b7-9763-b5ba26d60cfd_isr_945869_52.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/dfd89499-cf54-48b7-9763-b5ba26d60cfd_isr_945869_52.stderr parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb5682f9c839936443e182b6c20d1621 with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash 7b1b948ae9dae59e8b8ade889d8f28ea parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 864 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/2cf7ab96-403c-4e6b-b4cb-33c57bd48bef_characterizeImage_945756_7.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/2cf7ab96-403c-4e6b-b4cb-33c57bd48bef_characterizeImage_945756_7.stderr parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/29848de2-11e2-445f-b0e6-be6201e607d9_isr_945756_7.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/29848de2-11e2-445f-b0e6-be6201e607d9_isr_945756_7.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e5626144ea023996f4e475d68e67b44 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash b6a54a3edc2c3139225dcefdd36342f4 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 865 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/4a125f34-3588-49eb-9a94-d33aa49b3ebb_calibrate_945869_4.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/4a125f34-3588-49eb-9a94-d33aa49b3ebb_calibrate_945869_4.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1e1beb6c-29ed-4b9e-aa14-4ab8e05e71e4_characterizeImage_945869_4.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1e1beb6c-29ed-4b9e-aa14-4ab8e05e71e4_characterizeImage_945869_4.stderr parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a25b5127ec4846857474650b4bda02b3 with result from task 814 parsl.dataflow.dflow DEBUG: Task 814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 815 has memoization hash 76395d6ff0085a9e69b0263ffa399467 parsl.dataflow.memoization INFO: Task 815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 866 parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/3d8e0d27-890b-4470-ae87-a041b21cd005_calibrate_945756_39.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/3d8e0d27-890b-4470-ae87-a041b21cd005_calibrate_945756_39.stderr parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/7d3cf7ac-88a3-4575-8499-3f5a03b83a36_characterizeImage_945756_39.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/7d3cf7ac-88a3-4575-8499-3f5a03b83a36_characterizeImage_945756_39.stderr parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 629d921aee606d83c41eed77fac244fd with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash 8c7cdc69c1e6371961261404429bd086 parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246284720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 867 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/dba511ff-4b87-43b2-8f83-532ca7446aa3_calibrate_945757_9.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/dba511ff-4b87-43b2-8f83-532ca7446aa3_calibrate_945757_9.stderr parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/fcd7a9a7-8b9a-44ff-bd3f-2888d8877434_characterizeImage_945757_9.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/fcd7a9a7-8b9a-44ff-bd3f-2888d8877434_characterizeImage_945757_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:54 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d909fa9b08d3835a41a437cc8d511f81 with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash 60acb854ccc46918bd076b1b5f6a582f parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 868 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/aa2a72a8-53f8-4385-a5fe-fc518613ae05_calibrate_945756_15.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/aa2a72a8-53f8-4385-a5fe-fc518613ae05_calibrate_945756_15.stderr parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/238d3a32-e75c-46e9-9c8e-373696c7dc25_characterizeImage_945756_15.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/238d3a32-e75c-46e9-9c8e-373696c7dc25_characterizeImage_945756_15.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ffb47421a1900e3327e440f4d0e4f87 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4404c18e-f6ec-454e-a7b5-a11516eaa5d2_calibrate_945757_62.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4404c18e-f6ec-454e-a7b5-a11516eaa5d2_calibrate_945757_62.stderr parsl.dataflow.dflow INFO: Task 1084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4a9767bc736c66489405a3aec05ba60 with result from task 1084 parsl.dataflow.dflow DEBUG: Task 1084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1085 has memoization hash debdaaa865624c75d5877fbc56747001 parsl.dataflow.memoization INFO: Task 1085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1085 try 0 launched on executor multi with executor id 869 parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/16dbe4cc-05d1-4bd9-805f-ba74f7b7d3f1_calibrate_945757_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/16dbe4cc-05d1-4bd9-805f-ba74f7b7d3f1_calibrate_945757_30.stderr parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/34685442-e428-4e5f-8158-ee9661a31b89_characterizeImage_945757_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/34685442-e428-4e5f-8158-ee9661a31b89_characterizeImage_945757_30.stderr parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ce7668ac4576d5fa4e84a073c62dda4 with result from task 808 parsl.dataflow.dflow DEBUG: Task 808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 809 has memoization hash 990a3ac9f8346597e515d5329aef6dd2 parsl.dataflow.memoization INFO: Task 809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246346c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 870 parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/6e396d61-2583-475c-b8b5-0902e98821a7_calibrate_945868_57.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/6e396d61-2583-475c-b8b5-0902e98821a7_calibrate_945868_57.stderr parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0ec401d0-05bf-41e8-9138-ec2d16aaad3f_characterizeImage_945868_57.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0ec401d0-05bf-41e8-9138-ec2d16aaad3f_characterizeImage_945868_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:55 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb58bfa58c77095466fcd662d95a8c14 with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash c5f2e535a1a1a7141b8baa906d73fbe3 parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462853a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 871 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/22d1d061-a587-4cc3-8506-6319e7ebe162_calibrate_945869_11.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/22d1d061-a587-4cc3-8506-6319e7ebe162_calibrate_945869_11.stderr parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/8da39226-0174-4b75-ad90-ac25fc7f185d_characterizeImage_945869_11.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/8da39226-0174-4b75-ad90-ac25fc7f185d_characterizeImage_945869_11.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cabb54369a8aa41d001e4fcda373413 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/bf115159-9fd7-40a0-8783-e2d1be586bfd_calibrate_945757_34.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/bf115159-9fd7-40a0-8783-e2d1be586bfd_calibrate_945757_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:55 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8618e084036e335760e6970f2c3e20d6 with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 73c2f09baf2e9a8d3d1b10a950ac0b84 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247996a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 872 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f13a45f9-7e40-43d6-8155-e1abcb04b82c_calibrate_945868_56.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f13a45f9-7e40-43d6-8155-e1abcb04b82c_calibrate_945868_56.stderr parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/305c67b2-59a3-43d0-a731-3b75c848f947_characterizeImage_945868_56.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/305c67b2-59a3-43d0-a731-3b75c848f947_characterizeImage_945868_56.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ad74fa4d5fdace4f00b73bc5c233924 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash a48ed67aaf639dca9a3ab88148afb560 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 873 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c6678285-664c-4f1d-89e2-7b793565b988_calibrate_945867_57.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c6678285-664c-4f1d-89e2-7b793565b988_calibrate_945867_57.stderr parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4cb75670-e905-49cf-9f6b-e6fe3664faed_characterizeImage_945867_57.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4cb75670-e905-49cf-9f6b-e6fe3664faed_characterizeImage_945867_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 305 parsl.dataflow.strategy DEBUG: Executor multi has 305 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:36:55 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:36:55 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:56 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b76c30be793c7dc418bd5171cc3ada7 with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash df89b8065fca401c95b985beb8e2ba19 parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246287060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 874 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8a446210-c90e-4609-bc59-b822d7c808b0_calibrate_945868_5.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8a446210-c90e-4609-bc59-b822d7c808b0_calibrate_945868_5.stderr parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/8550d854-c3cb-4b25-90f7-da81a74bea6d_characterizeImage_945868_5.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/8550d854-c3cb-4b25-90f7-da81a74bea6d_characterizeImage_945868_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:56 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:56 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:56 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43ef125382346a36ee9901e3a5a0a561 with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c7364b65-81fd-43c6-962b-33981f93ac01_calibrate_945756_53.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c7364b65-81fd-43c6-962b-33981f93ac01_calibrate_945756_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:57 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8aff4ee8e5580b21679a38d29d6590cd with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/599b0918-0c0c-4493-9bc0-bef6f18c0c29_calibrate_945756_56.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/599b0918-0c0c-4493-9bc0-bef6f18c0c29_calibrate_945756_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:57 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 1035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4da60e7a658b2fbb2054fe9c6feeb32e with result from task 1035 parsl.dataflow.dflow DEBUG: Task 1035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash 5b6033948fa69de29e5ca8802d7e55f3 parsl.dataflow.memoization INFO: Task 1036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246089c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 875 parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/e72b5611-0055-47de-b273-203c6208d345_characterizeImage_945868_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/e72b5611-0055-47de-b273-203c6208d345_characterizeImage_945868_42.stderr parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c2a962f7-307d-47cb-9582-e6280183b0ea_isr_945868_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c2a962f7-307d-47cb-9582-e6280183b0ea_isr_945868_42.stderr parsl.dataflow.dflow INFO: Task 1026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e16079cc2d55c47b931e0fa918643267 with result from task 1026 parsl.dataflow.dflow DEBUG: Task 1026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash 84479d27753fbbb84f6f92dbba4fe0c1 parsl.dataflow.memoization INFO: Task 1027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246089300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 876 parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/3de3f01c-e506-4de1-8690-adf935e10122_characterizeImage_945755_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/3de3f01c-e506-4de1-8690-adf935e10122_characterizeImage_945755_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/c53b9597-9b1c-41f1-b8f3-9c2b414a7faa_isr_945755_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/c53b9597-9b1c-41f1-b8f3-9c2b414a7faa_isr_945755_16.stderr parsl.dataflow.dflow INFO: Task 1023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c38659664488e96a7db0a82970e4b7f0 with result from task 1023 parsl.dataflow.dflow DEBUG: Task 1023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash 069fba163299b9017fde506024a2ff3e parsl.dataflow.memoization INFO: Task 1024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246088fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 877 parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/914fd3e6-6a10-4c14-a6d6-f392dc1e149e_characterizeImage_945756_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/914fd3e6-6a10-4c14-a6d6-f392dc1e149e_characterizeImage_945756_45.stderr parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/dae71855-2081-4e23-8dad-5c91abe22e0d_isr_945756_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/dae71855-2081-4e23-8dad-5c91abe22e0d_isr_945756_45.stderr parsl.dataflow.dflow INFO: Task 1029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57c96b46e607b65227e1db6320df2828 with result from task 1029 parsl.dataflow.dflow DEBUG: Task 1029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash 6b08bd2e6107341c5f6499d99d72ae73 parsl.dataflow.memoization INFO: Task 1030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246089620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 878 parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dd804a8e-6844-41c7-b599-eb37eee39a0c_characterizeImage_945757_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dd804a8e-6844-41c7-b599-eb37eee39a0c_characterizeImage_945757_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/4a6ad8dc-1cb3-4e9d-bc8d-57deffe61a6a_isr_945757_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/4a6ad8dc-1cb3-4e9d-bc8d-57deffe61a6a_isr_945757_16.stderr parsl.dataflow.dflow INFO: Task 1044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e83835986358a5764de7380d38cff0b0 with result from task 1044 parsl.dataflow.dflow DEBUG: Task 1044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash e458afb1d4594504633d5c8e0378fa77 parsl.dataflow.memoization INFO: Task 1045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 879 parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/77384002-441d-4fdf-b778-4a370c0d4533_characterizeImage_945756_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/77384002-441d-4fdf-b778-4a370c0d4533_characterizeImage_945756_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/efe70310-0635-4d4d-b5f0-66520b2ca4e2_isr_945756_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/efe70310-0635-4d4d-b5f0-66520b2ca4e2_isr_945756_16.stderr parsl.dataflow.dflow INFO: Task 1038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 972eb685525ce8bd72239dd912fd916c with result from task 1038 parsl.dataflow.dflow DEBUG: Task 1038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash 8c5813b3bc0be26948594abe53e6d4dc parsl.dataflow.memoization INFO: Task 1039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246089f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 880 parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/a28c361b-ba1c-461c-a8b9-5b791fe4c479_characterizeImage_945867_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/a28c361b-ba1c-461c-a8b9-5b791fe4c479_characterizeImage_945867_52.stderr parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/ace99fde-37ab-490c-912e-9696217a36f1_isr_945867_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/ace99fde-37ab-490c-912e-9696217a36f1_isr_945867_52.stderr parsl.dataflow.dflow INFO: Task 1032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67f3a2ff71fffe9572ac875c89aaccab with result from task 1032 parsl.dataflow.dflow DEBUG: Task 1032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash 38fe8e63618a5530d5780c3696dc40fa parsl.dataflow.memoization INFO: Task 1033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246089940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 881 parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/de5cbb61-fc15-4e55-a6b4-5d02bfae531f_characterizeImage_945757_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/de5cbb61-fc15-4e55-a6b4-5d02bfae531f_characterizeImage_945757_13.stderr parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/4d5b8aa0-5056-458c-a065-cd1c4f0a5afd_isr_945757_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/4d5b8aa0-5056-458c-a065-cd1c4f0a5afd_isr_945757_13.stderr parsl.dataflow.dflow INFO: Task 1041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddaeed9a1d28569f39a6348910064ea6 with result from task 1041 parsl.dataflow.dflow DEBUG: Task 1041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash 5b247f85ebd3b629feea7fbc751b5d17 parsl.dataflow.memoization INFO: Task 1042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 882 parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0ee5ae70-4210-4084-9505-28ad3d6f9a1e_characterizeImage_945868_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0ee5ae70-4210-4084-9505-28ad3d6f9a1e_characterizeImage_945868_30.stderr parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c0a6d980-cb42-4512-a08d-7ab3f2a606c2_isr_945868_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c0a6d980-cb42-4512-a08d-7ab3f2a606c2_isr_945868_30.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb366492bc0585715a2fa73bea9f2f7f with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 2e4bf46307f7d7a49a8b8a77f9c2b9cd parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 883 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/b5eb9ee3-575f-4c62-8a00-3793c5f7580b_calibrate_945757_42.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/b5eb9ee3-575f-4c62-8a00-3793c5f7580b_calibrate_945757_42.stderr parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dc81e187-8481-4f76-8629-056506af64b2_characterizeImage_945757_42.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dc81e187-8481-4f76-8629-056506af64b2_characterizeImage_945757_42.stderr parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c69737ae32e4ee942c169eb9d4b1e69 with result from task 880 parsl.dataflow.dflow DEBUG: Task 880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 881 has memoization hash 355c1040bca6a8385bff1380a7ef9bd1 parsl.dataflow.memoization INFO: Task 881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461ab740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 884 parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/3817ff25-fb82-4c67-9c9b-d517fbd019b9_calibrate_945869_39.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/3817ff25-fb82-4c67-9c9b-d517fbd019b9_calibrate_945869_39.stderr parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/626a2605-7b8b-4545-a657-b7a174cf891b_characterizeImage_945869_39.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/626a2605-7b8b-4545-a657-b7a174cf891b_characterizeImage_945869_39.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f9fccf85185807c513cf23f1b7cc1b5 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 62e16edf017df1672226ecf66e987af1 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479165c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 885 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/500c4347-44a2-4097-8b0b-f21fea45d785_calibrate_945868_4.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/500c4347-44a2-4097-8b0b-f21fea45d785_calibrate_945868_4.stderr parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/56d86744-beed-4768-9cfa-eafe88c2595a_characterizeImage_945868_4.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/56d86744-beed-4768-9cfa-eafe88c2595a_characterizeImage_945868_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:57 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11e1fd39ec42457748fdc4a33474ad19 with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/41094bfc-5685-4a86-a01a-94dbf9b6eb26_calibrate_945756_8.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/41094bfc-5685-4a86-a01a-94dbf9b6eb26_calibrate_945756_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:58 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:58 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e41c2c20d6bdeb7cec45f14f2c7a55b with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/794551a9-77c5-42a5-b9e2-1181b6d3dfb0_calibrate_945757_26.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/794551a9-77c5-42a5-b9e2-1181b6d3dfb0_calibrate_945757_26.stderr parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de12a389fb6808c1e368e6f5540c5bb8 with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash 05388d66d1348138818d5c5a994e6b8a parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 886 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/3b06709d-317e-41f0-8eda-12105b4ee6b6_calibrate_945755_53.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/3b06709d-317e-41f0-8eda-12105b4ee6b6_calibrate_945755_53.stderr parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ccb92111-74ea-43f4-8251-0bfcddb30277_characterizeImage_945755_53.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/ccb92111-74ea-43f4-8251-0bfcddb30277_characterizeImage_945755_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:58 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c42cc810eff1b59ba52f944951a0c52 with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash d123e17e4a1d1b285a76fe4af7bffe08 parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246287ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 887 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79e8bb19-eaa6-4668-aeb3-7fd637994b80_calibrate_945868_36.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79e8bb19-eaa6-4668-aeb3-7fd637994b80_calibrate_945868_36.stderr parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/fe524c3b-a8e4-4124-bb30-deaf33fbb7ee_characterizeImage_945868_36.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/fe524c3b-a8e4-4124-bb30-deaf33fbb7ee_characterizeImage_945868_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:58 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 467131ae50d343ed807eab0cec46d99e with result from task 865 parsl.dataflow.dflow DEBUG: Task 865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 866 has memoization hash dc07d7138e880fed30bf241a523a21cc parsl.dataflow.memoization INFO: Task 866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aa7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 888 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/e36c9c77-2235-4e69-8bef-75bdd9ca8168_calibrate_945868_54.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/e36c9c77-2235-4e69-8bef-75bdd9ca8168_calibrate_945868_54.stderr parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/fbdc54e8-c221-4b77-a743-ad6749aa42d8_characterizeImage_945868_54.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/fbdc54e8-c221-4b77-a743-ad6749aa42d8_characterizeImage_945868_54.stderr parsl.dataflow.dflow INFO: Task 955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0ebff53a5c0a35ff550a8bc14442f03 with result from task 955 parsl.dataflow.dflow DEBUG: Task 955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 956 has memoization hash 36cd0cfafc54676d9cb9254dd9c0d961 parsl.dataflow.memoization INFO: Task 956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 889 parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/52c38800-3593-4344-94c6-94ad3e91577e_calibrate_945755_54.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/52c38800-3593-4344-94c6-94ad3e91577e_calibrate_945755_54.stderr parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/0048233f-a093-4d88-ad51-ddd5defcfca0_characterizeImage_945755_54.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/0048233f-a093-4d88-ad51-ddd5defcfca0_characterizeImage_945755_54.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 013be670eebf8f81dc114239ee55e185 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 6e133d6fa01181938c29b2c1a4f2ace1 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247916c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:59 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 890 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/e8cf0312-c471-4c5f-9c62-fae4e7b2eac2_calibrate_945869_62.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/e8cf0312-c471-4c5f-9c62-fae4e7b2eac2_calibrate_945869_62.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2092bf89-0583-4ffa-84b0-5b1418e1983b_characterizeImage_945869_62.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2092bf89-0583-4ffa-84b0-5b1418e1983b_characterizeImage_945869_62.stderr parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bde5d3607e215eed193b9122ef1ae327 with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash fcaa17cf1a301b25afed06dfb8c5e357 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462be200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 891 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b8098a9d-8d98-4b2f-9bb4-6c078a1f1de6_calibrate_945869_31.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b8098a9d-8d98-4b2f-9bb4-6c078a1f1de6_calibrate_945869_31.stderr parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/ec019aa9-3ea6-4410-a5e6-1b7cf7a30ba8_characterizeImage_945869_31.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/ec019aa9-3ea6-4410-a5e6-1b7cf7a30ba8_characterizeImage_945869_31.stderr parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bba07fa05c518b04f91afbb7b680eec5 with result from task 857 parsl.dataflow.dflow DEBUG: Task 857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/bebf7275-802f-4ee4-bf0b-f5e609f0faef_calibrate_945755_14.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/bebf7275-802f-4ee4-bf0b-f5e609f0faef_calibrate_945755_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:59 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 603b3ffe86fd527c1f763b708d6f81a2 with result from task 871 parsl.dataflow.dflow DEBUG: Task 871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 872 has memoization hash b709c528232012d6472ec7a4bf24037f parsl.dataflow.memoization INFO: Task 872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 892 parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/7685a7a1-c1d8-4cc5-be45-6d0f728ee91d_calibrate_945867_26.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/7685a7a1-c1d8-4cc5-be45-6d0f728ee91d_calibrate_945867_26.stderr parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/1e5d8cd9-7ab9-485b-aa50-04393093bea5_characterizeImage_945867_26.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/1e5d8cd9-7ab9-485b-aa50-04393093bea5_characterizeImage_945867_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:36:59 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfbeba4c6a1ed4f6e8a24690720face3 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/e3850802-3eff-44ec-8c1f-88bf917391b3_calibrate_945757_8.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/e3850802-3eff-44ec-8c1f-88bf917391b3_calibrate_945757_8.stderr parsl.dataflow.dflow INFO: Task 1112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe950f75ea300015c1b397b5b6556b86 with result from task 1112 parsl.dataflow.dflow DEBUG: Task 1112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d3f9e3b3-b2ee-4017-b90e-cc677d163c0d_calibrate_945755_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d3f9e3b3-b2ee-4017-b90e-cc677d163c0d_calibrate_945755_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:00 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec86d12a9e6cc3bad335fbce025a81b2 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 212b4c51b323b73fdaf9cc98deb1cd59 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 893 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2aa80c99-ae0a-4aed-b685-3bbdb9ce45f9_calibrate_945868_23.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2aa80c99-ae0a-4aed-b685-3bbdb9ce45f9_calibrate_945868_23.stderr parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/b0d7e9d4-e557-420a-a042-3b70e4c4ad08_characterizeImage_945868_23.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/b0d7e9d4-e557-420a-a042-3b70e4c4ad08_characterizeImage_945868_23.stderr parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e340444fac2be8aa3934005571e62633 with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash cd6f7289c151b0b8efd928a5c0ee50f8 parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246286160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 894 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/d8976979-1c96-4ef4-a8dc-6eb4f6146e70_calibrate_945756_18.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/d8976979-1c96-4ef4-a8dc-6eb4f6146e70_calibrate_945756_18.stderr parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/3ee8d640-819e-484d-ba4d-52b2fc118b31_characterizeImage_945756_18.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/3ee8d640-819e-484d-ba4d-52b2fc118b31_characterizeImage_945756_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:00 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41172e8940b9f2169e1cddd299a835c7 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 0efb2bbb2405de08aa221e267948f560 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 895 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/89cf6cef-56fd-4fbe-bb9e-1bcb8d07f61b_calibrate_945757_15.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/89cf6cef-56fd-4fbe-bb9e-1bcb8d07f61b_calibrate_945757_15.stderr parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/cd3ef40f-4bc5-48dc-bd2e-477b2a32df19_characterizeImage_945757_15.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/cd3ef40f-4bc5-48dc-bd2e-477b2a32df19_characterizeImage_945757_15.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6eee79fb1eae26402ed57172610a567e with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/dacf3ba3-4a84-4ebf-87e4-689ebb7e0b52_calibrate_945757_59.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/dacf3ba3-4a84-4ebf-87e4-689ebb7e0b52_calibrate_945757_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:00 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e363d7628c9880d2c8952426051c1d03 with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/b9645bcb-c7c8-4bbb-b123-d407ec7a9f19_calibrate_945756_22.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/b9645bcb-c7c8-4bbb-b123-d407ec7a9f19_calibrate_945756_22.stderr parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 625b0f577c73c040039b5850cf30fc2a with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/2815e669-50aa-4517-91ff-95dc185287fb_calibrate_945757_3.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/2815e669-50aa-4517-91ff-95dc185287fb_calibrate_945757_3.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ac172261c04e9ddf3199f04f6ef3e3c with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 9488a0a58b87ff57591f6b157e0f4a62 parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 896 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/7fd7e4c5-c5d9-49e9-8392-c3b5f25a4138_characterizeImage_945867_14.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/7fd7e4c5-c5d9-49e9-8392-c3b5f25a4138_characterizeImage_945867_14.stderr parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/332adf15-eee1-4bfa-aa10-d6f3f70f7b40_isr_945867_14.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/332adf15-eee1-4bfa-aa10-d6f3f70f7b40_isr_945867_14.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60a0ad819da6a5430430433a7b2e3736 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 12ddc4fb71cd7d1cae3ac97fbe30f680 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 897 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/6a75f1ae-9a2e-4cff-a581-9c1d5da7daa1_characterizeImage_945868_47.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/6a75f1ae-9a2e-4cff-a581-9c1d5da7daa1_characterizeImage_945868_47.stderr parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c589d8cd-717a-4efe-824f-6b9812178af9_isr_945868_47.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/c589d8cd-717a-4efe-824f-6b9812178af9_isr_945868_47.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47170bfe6a3cff3eb0a571da283c1b7a with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash bd520330ed4e3f7a3505bd1a52fcb386 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 898 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/8490f1af-9c55-4453-90dd-2f5fd8983244_characterizeImage_945867_59.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/8490f1af-9c55-4453-90dd-2f5fd8983244_characterizeImage_945867_59.stderr parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4c58c91d-ef29-4930-9fe7-4832ba83a993_isr_945867_59.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/4c58c91d-ef29-4930-9fe7-4832ba83a993_isr_945867_59.stderr parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e9424c99046dea24a793262dc3cbf49 with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 422c6bce204de6a3bbd00a86368ea6b8 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:01 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 899 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/e0d88d16-8b49-4665-886f-1ddd1cd122bb_characterizeImage_945756_34.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/e0d88d16-8b49-4665-886f-1ddd1cd122bb_characterizeImage_945756_34.stderr parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/580272e9-d943-4f2f-b557-1c299fb1f992_isr_945756_34.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/580272e9-d943-4f2f-b557-1c299fb1f992_isr_945756_34.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6274096010e0334450993a4bb896d239 with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash a85a16c49066520d3da3d1f7c1ac3182 parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 900 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2d027c1f-06d0-4a7e-a084-6d4c950bd09d_characterizeImage_945869_49.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2d027c1f-06d0-4a7e-a084-6d4c950bd09d_characterizeImage_945869_49.stderr parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/abc00f5e-a860-46d1-ae98-47416992e45c_isr_945869_49.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/abc00f5e-a860-46d1-ae98-47416992e45c_isr_945869_49.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d21761e42fddc5a30502c43b5fa9d45 with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 78fc6194dea4e180526fce26cf8bdb3d parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 901 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/05b593f7-f5f2-4a14-bb29-93327950c08f_characterizeImage_945755_38.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/05b593f7-f5f2-4a14-bb29-93327950c08f_characterizeImage_945755_38.stderr parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8b35d30e-8a67-431c-8071-235cf3a7e210_isr_945755_38.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/8b35d30e-8a67-431c-8071-235cf3a7e210_isr_945755_38.stderr parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 113f1d838abcbf1b226503f3336d4e48 with result from task 860 parsl.dataflow.dflow DEBUG: Task 860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/20dab909-0299-468c-b217-6b1777232041_calibrate_945755_40.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/20dab909-0299-468c-b217-6b1777232041_calibrate_945755_40.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d2eeb056d22d7eb9578f32b6ec48541 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 1c456671593a99ee7d286be7cff60161 parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 902 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/7196bb14-a6d5-41c8-9022-7cb1276bef4a_characterizeImage_945756_48.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/7196bb14-a6d5-41c8-9022-7cb1276bef4a_characterizeImage_945756_48.stderr parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/3c9226f4-5f46-4346-89b5-d36aa7c530ea_isr_945756_48.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/3c9226f4-5f46-4346-89b5-d36aa7c530ea_isr_945756_48.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ded97aa74dbf3f1158d9e0b06cb1648a with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash f5c478502d9b11e0497907d5d5a4d102 parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 903 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/faf4035b-3520-4717-a863-3e67a2e2f90a_characterizeImage_945867_41.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/faf4035b-3520-4717-a863-3e67a2e2f90a_characterizeImage_945867_41.stderr parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/ab6d557c-5609-41e6-b7b2-3126f187fbba_isr_945867_41.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/ab6d557c-5609-41e6-b7b2-3126f187fbba_isr_945867_41.stderr parsl.dataflow.dflow INFO: Task 1014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 444a5147c05bd67eb08672fa62a9c889 with result from task 1014 parsl.dataflow.dflow DEBUG: Task 1014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash e5adc191007432261500d178234ca55c parsl.dataflow.memoization INFO: Task 1015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246088680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 904 parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/33d5e6ea-a671-4801-bb60-c83fe063bda0_characterizeImage_945756_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/33d5e6ea-a671-4801-bb60-c83fe063bda0_characterizeImage_945756_27.stderr parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/a3c0a83e-640a-4e4b-a56a-ce080ba3cccf_isr_945756_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/a3c0a83e-640a-4e4b-a56a-ce080ba3cccf_isr_945756_27.stderr parsl.dataflow.dflow INFO: Task 999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3239628755f06bcb82ee5fbc62f054f with result from task 999 parsl.dataflow.dflow DEBUG: Task 999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash 5c84f3a8d5aed1a5a2b1123fbc913d37 parsl.dataflow.memoization INFO: Task 1000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 905 parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/41ba1c95-4313-4985-a4dd-9525d99cbfca_characterizeImage_945755_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/41ba1c95-4313-4985-a4dd-9525d99cbfca_characterizeImage_945755_33.stderr parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/1a2961e1-257d-4e03-b212-de2c8d8c8f21_isr_945755_33.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/1a2961e1-257d-4e03-b212-de2c8d8c8f21_isr_945755_33.stderr parsl.dataflow.dflow INFO: Task 1011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08fd5dc91dd33bd84e4ae4fd01c9a8d8 with result from task 1011 parsl.dataflow.dflow DEBUG: Task 1011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash 3d28fc721310b1dfae25638e03c16c8a parsl.dataflow.memoization INFO: Task 1012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246088360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 906 parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ec6d412e-73c8-4b7c-8ee8-386f556a4afb_characterizeImage_945867_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ec6d412e-73c8-4b7c-8ee8-386f556a4afb_characterizeImage_945867_19.stderr parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/31ce2812-944b-4704-bdb4-4600e1294722_isr_945867_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/31ce2812-944b-4704-bdb4-4600e1294722_isr_945867_19.stderr parsl.dataflow.dflow INFO: Task 1017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9331b7b8d0b66809a6f39447896aac5 with result from task 1017 parsl.dataflow.dflow DEBUG: Task 1017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash d25570846ac3a6847d829347dc4ef3f0 parsl.dataflow.memoization INFO: Task 1018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460889a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 907 parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/d895e69a-5887-4611-93f2-02f0bb7c2ace_characterizeImage_945867_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/d895e69a-5887-4611-93f2-02f0bb7c2ace_characterizeImage_945867_35.stderr parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/19c80da6-eb3e-402e-901d-4c1d3ae18dde_isr_945867_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/19c80da6-eb3e-402e-901d-4c1d3ae18dde_isr_945867_35.stderr parsl.dataflow.dflow INFO: Task 1020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e295115ae332f3853be5f61e3d2f7129 with result from task 1020 parsl.dataflow.dflow DEBUG: Task 1020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash 9420499e591ba997fbcf8786a6b54635 parsl.dataflow.memoization INFO: Task 1021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246088cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 908 parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/dd53d6b7-31a9-4509-a0e4-d28e512dee5c_characterizeImage_945868_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/dd53d6b7-31a9-4509-a0e4-d28e512dee5c_characterizeImage_945868_39.stderr parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/98aff5f1-455c-4267-b6c4-91f55a0eebfc_isr_945868_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/98aff5f1-455c-4267-b6c4-91f55a0eebfc_isr_945868_39.stderr parsl.dataflow.dflow INFO: Task 1002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9b92202117138a22a933bc83c907b5c with result from task 1002 parsl.dataflow.dflow DEBUG: Task 1002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash d2d02c80f440deb94857eaac39a97e32 parsl.dataflow.memoization INFO: Task 1003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 909 parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/02796bba-c3fd-4d86-9587-2a50638d7315_characterizeImage_945756_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/02796bba-c3fd-4d86-9587-2a50638d7315_characterizeImage_945756_20.stderr parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2dd1616f-439c-4a8c-820d-148a470e52d6_isr_945756_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/2dd1616f-439c-4a8c-820d-148a470e52d6_isr_945756_20.stderr parsl.dataflow.dflow INFO: Task 1008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b46c3d91b2a2e3e6364698769019cf02 with result from task 1008 parsl.dataflow.dflow DEBUG: Task 1008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash 971aa82212cfbe5f267bae096c11b9d0 parsl.dataflow.memoization INFO: Task 1009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 910 parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f06fa351-9a99-482e-a544-1c019d6afbe7_characterizeImage_945756_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f06fa351-9a99-482e-a544-1c019d6afbe7_characterizeImage_945756_25.stderr parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/d362da4e-9cac-4e75-948e-f5b65028536d_isr_945756_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/d362da4e-9cac-4e75-948e-f5b65028536d_isr_945756_25.stderr parsl.dataflow.dflow INFO: Task 1005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 263664773e263b50dc0b0b172227e347 with result from task 1005 parsl.dataflow.dflow DEBUG: Task 1005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash 2f2deb96604ba65551005ab6cb8c0477 parsl.dataflow.memoization INFO: Task 1006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 911 parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/f9cd966e-24a3-493c-a033-5b1d35811293_characterizeImage_945869_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/f9cd966e-24a3-493c-a033-5b1d35811293_characterizeImage_945869_59.stderr parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/922a3708-c88d-40da-9a1c-e12cf1ff9052_isr_945869_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/922a3708-c88d-40da-9a1c-e12cf1ff9052_isr_945869_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:01 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e5e315cf9720c1cdc859137144ad69a with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7868f511-4798-4b4b-a256-dd9b6edb84cd_calibrate_945755_42.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7868f511-4798-4b4b-a256-dd9b6edb84cd_calibrate_945755_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:01 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39cc9ecc58055232dfae4c72fa6d8004 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 37849c912ccad83384d1609034a29f61 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 912 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c88f99dd-765b-485a-97ed-b7a7a62f6e4c_calibrate_945867_21.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c88f99dd-765b-485a-97ed-b7a7a62f6e4c_calibrate_945867_21.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4552cfe4-9246-4a5f-9e8a-84d939b182ff_characterizeImage_945867_21.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4552cfe4-9246-4a5f-9e8a-84d939b182ff_characterizeImage_945867_21.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea6bc891a2576428696bf27f2d063e0d with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash c15e24ec3dae566aed89b95b9ca7a755 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 913 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/206e2b22-6331-4304-824e-f3fe1bc0bb2a_calibrate_945869_8.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/206e2b22-6331-4304-824e-f3fe1bc0bb2a_calibrate_945869_8.stderr parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/4633bbac-56c1-4997-897d-d025a7f50dd4_characterizeImage_945869_8.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/4633bbac-56c1-4997-897d-d025a7f50dd4_characterizeImage_945869_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:01 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5432275f2365fcde34a580f459f551fb with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 1392c9abbbf74d9fcd80fd55f10ae36d parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247997380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 914 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/15b627c3-7203-4dbe-92da-1abd9a803bab_calibrate_945756_19.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/15b627c3-7203-4dbe-92da-1abd9a803bab_calibrate_945756_19.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/657eae25-6289-49b1-9dcd-96c39e272493_characterizeImage_945756_19.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/657eae25-6289-49b1-9dcd-96c39e272493_characterizeImage_945756_19.stderr parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e76f89fa85c1e1ae95e757ae5a37e85 with result from task 746 parsl.dataflow.dflow DEBUG: Task 746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8b886367-c0ff-48b5-a6b3-71d4747d2212_calibrate_945756_52.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8b886367-c0ff-48b5-a6b3-71d4747d2212_calibrate_945756_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:02 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:02 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3692aafd50f95bcba72e2d0cda1b420d with result from task 958 parsl.dataflow.dflow DEBUG: Task 958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 959 has memoization hash e636a591b47730b0482592c428d86ae5 parsl.dataflow.memoization INFO: Task 959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 959 try 0 launched on executor multi with executor id 915 parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/f576f2a4-8703-4368-b2b0-870d97e00ebf_calibrate_945756_23.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/f576f2a4-8703-4368-b2b0-870d97e00ebf_calibrate_945756_23.stderr parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/419e4477-d32e-492a-8420-11a76d73df8a_characterizeImage_945756_23.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/419e4477-d32e-492a-8420-11a76d73df8a_characterizeImage_945756_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:02 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 1094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 834d0f4f464805abe05d5020d15ae514 with result from task 1094 parsl.dataflow.dflow DEBUG: Task 1094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/f9cc263d-d182-49f6-8c8b-969a697fecf9_calibrate_945757_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/f9cc263d-d182-49f6-8c8b-969a697fecf9_calibrate_945757_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:03 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:03 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73c5b540685f22d95ef944d7c63506d7 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash ef3656a9b216e74afcdb67120469463c parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247917100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 916 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8e306d81-f97f-43ce-bb9b-12cf9bc33f15_calibrate_945868_35.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8e306d81-f97f-43ce-bb9b-12cf9bc33f15_calibrate_945868_35.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/6bbcf4dd-bab9-4665-bc88-e33fddd327ab_characterizeImage_945868_35.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/6bbcf4dd-bab9-4665-bc88-e33fddd327ab_characterizeImage_945868_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:03 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d79fac471875bf7986e3c531fa5ff1e with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash dea282f8053e32353ca1b4c4de56bc19 parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462858a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 917 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d6e929af-17b5-4887-ba7e-d19f714bca29_calibrate_945755_29.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d6e929af-17b5-4887-ba7e-d19f714bca29_calibrate_945755_29.stderr parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/748ebc48-b4ce-470a-a052-078555c3b7f8_characterizeImage_945755_29.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/748ebc48-b4ce-470a-a052-078555c3b7f8_characterizeImage_945755_29.stderr parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 908dd02ad8cba23ae499161e860c99c2 with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash 9086faebb450885ba1d3c50b3bd23a6b parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246284400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 918 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d3da880c-83a6-4aa3-8f63-7f3f830a8d53_calibrate_945757_31.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d3da880c-83a6-4aa3-8f63-7f3f830a8d53_calibrate_945757_31.stderr parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/32dbcdfd-02c8-4175-9d5b-1870ba78a03b_characterizeImage_945757_31.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/32dbcdfd-02c8-4175-9d5b-1870ba78a03b_characterizeImage_945757_31.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb794071255e50e53307ad1fa35d1d9a with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4e1a3d47-012f-415e-b712-67212a15891b_calibrate_945757_57.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4e1a3d47-012f-415e-b712-67212a15891b_calibrate_945757_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:04 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ba4ba88ee730a79b6d871ca04b60824 with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash d2f3c8ed0febd09603613fd7f8dfdf00 parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246310fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 919 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/dc32c03b-2834-4644-9c2b-b73b8e1569c7_calibrate_945869_3.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/dc32c03b-2834-4644-9c2b-b73b8e1569c7_calibrate_945869_3.stderr parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/9cc09445-898c-457f-9698-ab4daf10eedc_characterizeImage_945869_3.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/9cc09445-898c-457f-9698-ab4daf10eedc_characterizeImage_945869_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:04 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 1067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9382be0d0f2e4e9abb7c715687fd0cd7 with result from task 1067 parsl.dataflow.dflow DEBUG: Task 1067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/263d0ba8-5c77-48b0-b234-cf6812eed249_calibrate_945757_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/263d0ba8-5c77-48b0-b234-cf6812eed249_calibrate_945757_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:04 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:04 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10e108d3730192717d1becb92943b071 with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/002c269d-5810-4e98-af3a-80ce8accb8e9_calibrate_945756_41.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/002c269d-5810-4e98-af3a-80ce8accb8e9_calibrate_945756_41.stderr parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a245db905477be0ae805fd179691e850 with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 1f654e312884e3948d6fbde78fd995bc parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246287a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 920 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c6385baa-c413-42cf-be51-d88062b561be_calibrate_945756_14.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c6385baa-c413-42cf-be51-d88062b561be_calibrate_945756_14.stderr parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/19bcabc0-ee98-439d-9cf7-df052131c5c8_characterizeImage_945756_14.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/19bcabc0-ee98-439d-9cf7-df052131c5c8_characterizeImage_945756_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:05 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b13657b411833cb6e79d4bf3e3e1518 with result from task 821 parsl.dataflow.dflow DEBUG: Task 821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/527221cc-7364-4bb8-afdd-920896b71071_calibrate_945756_10.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/527221cc-7364-4bb8-afdd-920896b71071_calibrate_945756_10.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 829ad22e6bc4ba543df740f9e1210187 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 60a8e5ae4d83bd435cc76c17b85512c4 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 921 parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/29fb3726-13b4-41b4-b86b-b3491f0de1bf_calibrate_945868_43.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/29fb3726-13b4-41b4-b86b-b3491f0de1bf_calibrate_945868_43.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/51c6be92-0fe3-4021-b662-6bf04ced4012_characterizeImage_945868_43.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/51c6be92-0fe3-4021-b662-6bf04ced4012_characterizeImage_945868_43.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 1114 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:05 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k parsl.dataflow.memoization DEBUG: Storing app cache entry b809ac1148d8a6a7aec879df83a5884f with result from task 1114 proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow DEBUG: Task 1114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1115 has memoization hash aaa7e872420b926aa2fcc3aa03b9f3d5 parsl.dataflow.memoization INFO: Task 1115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460beb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1115 try 0 launched on executor multi with executor id 922 parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f614a9ae-e880-4b24-92d0-0198983d2d08_calibrate_945867_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f614a9ae-e880-4b24-92d0-0198983d2d08_calibrate_945867_50.stderr parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2502c862-aa77-427c-bc14-a75a9ee3e910_characterizeImage_945867_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2502c862-aa77-427c-bc14-a75a9ee3e910_characterizeImage_945867_50.stderr parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 764ed665ab2a8c39e3bb04cf980e07d3 with result from task 790 parsl.dataflow.dflow DEBUG: Task 790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash 4fcf154a8c3fa6d3aa45e22a8f82c3c8 parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246345940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 923 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/0633eb18-9bfb-4124-9e71-71e8e5528ad1_calibrate_945756_1.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/0633eb18-9bfb-4124-9e71-71e8e5528ad1_calibrate_945756_1.stderr parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/a0cf1cef-8519-48f8-816d-08c54de7e0a7_characterizeImage_945756_1.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/a0cf1cef-8519-48f8-816d-08c54de7e0a7_characterizeImage_945756_1.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 610fa1b9306238208c69e8b7607f5f94 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/415c1617-bcbe-4b6c-a69b-82a311dab4c7_calibrate_945757_5.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/415c1617-bcbe-4b6c-a69b-82a311dab4c7_calibrate_945757_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:05 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1b026fc8f93220f0a08b67498b2da19 with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/544cb80c-b9ac-437a-a8c7-8aca24a23dad_calibrate_945757_11.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/544cb80c-b9ac-437a-a8c7-8aca24a23dad_calibrate_945757_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:06 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6a035983b64891ab1ef02d6a1fbfb7a with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 0d76ac2f13d8970266d0abc4a3dc6c07 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247915da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 924 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/95678e94-3e5f-4805-804a-55a1881d9410_calibrate_945868_11.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/95678e94-3e5f-4805-804a-55a1881d9410_calibrate_945868_11.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/4c493b7f-5700-40a7-85e3-98bdd8a5988e_characterizeImage_945868_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/4c493b7f-5700-40a7-85e3-98bdd8a5988e_characterizeImage_945868_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:06 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:06 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:07 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:07 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 478cd3a3844a59446b849ac7f8814aed with result from task 921 parsl.dataflow.dflow DEBUG: Task 921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash 0977e5a6965281a564153c961e7c2cda parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 925 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6759b505-bf94-47cd-8bac-678eea613eb7_characterizeImage_945756_3.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6759b505-bf94-47cd-8bac-678eea613eb7_characterizeImage_945756_3.stderr parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/61f1264b-9fd9-4e16-87dc-828b958f3dc4_isr_945756_3.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/61f1264b-9fd9-4e16-87dc-828b958f3dc4_isr_945756_3.stderr parsl.dataflow.dflow INFO: Task 933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0761ec6bdd7ccb8df884879c41507a21 with result from task 933 parsl.dataflow.dflow DEBUG: Task 933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 934 has memoization hash 4f1a576176e7cb51aebaedb6bb74003a parsl.dataflow.memoization INFO: Task 934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246228c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 926 parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/3e11dade-18b6-4033-9c02-ce2a97630e34_characterizeImage_945756_6.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/3e11dade-18b6-4033-9c02-ce2a97630e34_characterizeImage_945756_6.stderr parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/8bbe791d-ca8b-4f5d-8bd3-86ba22a7f38e_isr_945756_6.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/8bbe791d-ca8b-4f5d-8bd3-86ba22a7f38e_isr_945756_6.stderr parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db25bae84f7cc0aee8d9d0815b4caa0d with result from task 927 parsl.dataflow.dflow DEBUG: Task 927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash a751ec6bc8fafd88ebe9d89b186b2427 parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246229260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 927 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/3b9d20fc-d300-45bf-800c-7986868df61f_characterizeImage_945757_44.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/3b9d20fc-d300-45bf-800c-7986868df61f_characterizeImage_945757_44.stderr parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/60c40a2e-6038-4969-8256-a5a65d1d7332_isr_945757_44.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/60c40a2e-6038-4969-8256-a5a65d1d7332_isr_945757_44.stderr parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8663c33e3bf2045d65bc9a358ac3b15 with result from task 915 parsl.dataflow.dflow DEBUG: Task 915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 916 has memoization hash 5e49299354b9f14ec5ec5d686d31a5ec parsl.dataflow.memoization INFO: Task 916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 928 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4998cc52-2253-47ff-bbcf-645062e9b75c_characterizeImage_945867_23.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4998cc52-2253-47ff-bbcf-645062e9b75c_characterizeImage_945867_23.stderr parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2f492838-739b-41e2-9cdf-062bf67a1f50_isr_945867_23.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/2f492838-739b-41e2-9cdf-062bf67a1f50_isr_945867_23.stderr parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0779bf9f895ee1490bbc97e7f9564d4 with result from task 912 parsl.dataflow.dflow DEBUG: Task 912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 913 has memoization hash b843b5d8d9b261775f1fb7b3a8d3203b parsl.dataflow.memoization INFO: Task 913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 929 parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/dd434d45-d0b9-4fa2-9312-6fdc14033cdf_characterizeImage_945869_18.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/dd434d45-d0b9-4fa2-9312-6fdc14033cdf_characterizeImage_945869_18.stderr parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f2a13a97-1718-4e8a-9d22-f0b1fd9fa1da_isr_945869_18.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/f2a13a97-1718-4e8a-9d22-f0b1fd9fa1da_isr_945869_18.stderr parsl.dataflow.dflow INFO: Task 930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32ef8fba6b716fc62707bc2bb1888159 with result from task 930 parsl.dataflow.dflow DEBUG: Task 930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 931 has memoization hash 29e90f41472b81dd37ea84c625c1260b parsl.dataflow.memoization INFO: Task 931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246228f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 930 parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/1f5b32d1-f002-4854-906f-f9337c678523_characterizeImage_945755_13.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/1f5b32d1-f002-4854-906f-f9337c678523_characterizeImage_945755_13.stderr parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/51fbc5f0-f4ae-4cf6-b957-4b79dcb8eecd_isr_945755_13.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/51fbc5f0-f4ae-4cf6-b957-4b79dcb8eecd_isr_945755_13.stderr parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80644e4ff1df35952cd75ace5f380b10 with result from task 924 parsl.dataflow.dflow DEBUG: Task 924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 925 has memoization hash 22ff4e8062edaa3dbf15ed768f2152a9 parsl.dataflow.memoization INFO: Task 925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246229760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 931 parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/7927cc51-8883-4440-9dcc-db626c6eb77e_characterizeImage_945868_28.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/7927cc51-8883-4440-9dcc-db626c6eb77e_characterizeImage_945868_28.stderr parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9602c900-dcd9-407c-a00c-5fa663d2328a_isr_945868_28.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/9602c900-dcd9-407c-a00c-5fa663d2328a_isr_945868_28.stderr parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bc4e4420a5ee0483ca398a6f557344c with result from task 918 parsl.dataflow.dflow DEBUG: Task 918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 919 has memoization hash c999a2caa8510f5286c865b12d87dd3b parsl.dataflow.memoization INFO: Task 919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 932 parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/4cb5e501-c590-4a87-aaa2-95eb8b16c3b7_characterizeImage_945755_44.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/4cb5e501-c590-4a87-aaa2-95eb8b16c3b7_characterizeImage_945755_44.stderr parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/2be981d6-5505-48cb-9489-639b4dd4100f_isr_945755_44.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/2be981d6-5505-48cb-9489-639b4dd4100f_isr_945755_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:07 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6136ae629d03253ed265920e17fd02e with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash 8cd49ee39b329dd62f8f26a8276dedc2 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246403ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 933 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/b018b229-3e5d-48f2-b019-278224ccb377_characterizeImage_945867_25.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/b018b229-3e5d-48f2-b019-278224ccb377_characterizeImage_945867_25.stderr parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/85c74c79-959c-47bc-8e17-adc11942655c_isr_945867_25.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/85c74c79-959c-47bc-8e17-adc11942655c_isr_945867_25.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a29fe4330693b2c2cad2aee6e8fba59c with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 9f0b7d4b611083665b48b0ae12972596 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 934 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/99ef614b-8bfb-44f6-9d52-ea0ede28e218_characterizeImage_945867_34.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/99ef614b-8bfb-44f6-9d52-ea0ede28e218_characterizeImage_945867_34.stderr parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/d01630ea-2d5d-452f-81af-57a2b04ef1d8_isr_945867_34.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/d01630ea-2d5d-452f-81af-57a2b04ef1d8_isr_945867_34.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03e3aaa86d6dc7bf1c69e893c6ceb9ff with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash 8273c81d809a4198e84d11cf87420f10 parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 935 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/2b27fb1e-ced2-43ae-93f7-547c73f4fcff_characterizeImage_945757_21.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/2b27fb1e-ced2-43ae-93f7-547c73f4fcff_characterizeImage_945757_21.stderr parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d2ccce26-978a-446e-b1a2-2ce8fe701fe0_isr_945757_21.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/d2ccce26-978a-446e-b1a2-2ce8fe701fe0_isr_945757_21.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cab6039322f793980a91cfaf53d92d85 with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 54379e0520a208cc2069d9cd2d3e5cb5 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 936 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/adeaca90-bd4c-4845-8619-13aa2c0abed2_characterizeImage_945755_12.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/adeaca90-bd4c-4845-8619-13aa2c0abed2_characterizeImage_945755_12.stderr parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/6da22f22-b0e6-4e1e-8905-ba2cb0e1fa7a_isr_945755_12.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/6da22f22-b0e6-4e1e-8905-ba2cb0e1fa7a_isr_945755_12.stderr parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42322d6ed2391737db665892b8b510bc with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash cbe49ba270a93f612fe24544eadc07b8 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246403f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 937 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/046cffaf-4595-4f4a-a83e-5b6b79d83d51_characterizeImage_945868_25.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/046cffaf-4595-4f4a-a83e-5b6b79d83d51_characterizeImage_945868_25.stderr parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/2a3e3c86-cea9-42ed-b2d6-7d1235febbeb_isr_945868_25.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/2a3e3c86-cea9-42ed-b2d6-7d1235febbeb_isr_945868_25.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fff4d5dc31a2e03831398539e5b9cbb with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash 42a92b71775f001ef49d0da0cd60abb5 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246403100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 938 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/0e11714c-a0ca-4da3-aaf5-42594f854e7e_characterizeImage_945756_43.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/0e11714c-a0ca-4da3-aaf5-42594f854e7e_characterizeImage_945756_43.stderr parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/fba0e134-a464-40a3-a369-a2d16ab125f1_isr_945756_43.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/fba0e134-a464-40a3-a369-a2d16ab125f1_isr_945756_43.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88719fe47282c5cc86b78f33b1305873 with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash ce0f7ffa88e68d17148e3440655f0927 parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246403ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 939 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bbcb88ca-1d01-4b2e-9050-e5d20dc0de14_characterizeImage_945869_16.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bbcb88ca-1d01-4b2e-9050-e5d20dc0de14_characterizeImage_945869_16.stderr parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/43e82438-4d83-4843-bf88-417af9b0792a_isr_945869_16.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/43e82438-4d83-4843-bf88-417af9b0792a_isr_945869_16.stderr parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd7b6d348e51dd31a9e19696d030c15f with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 9e50437b61e8c7cc20fa51e044f6252b parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246402840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 940 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/7c2c1d31-49c1-428f-b8ea-e6ecc4877418_characterizeImage_945869_22.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/7c2c1d31-49c1-428f-b8ea-e6ecc4877418_characterizeImage_945869_22.stderr parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/41ae89f9-fa86-4190-9d80-f101343326af_isr_945869_22.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/41ae89f9-fa86-4190-9d80-f101343326af_isr_945869_22.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b41b50cb66ff9cf7598a277702787e7c with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash cd247231f655fa64222d64cc27ce9a0e parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246403740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 941 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4607e1c1-2060-4625-9911-452b33744ea8_characterizeImage_945756_44.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4607e1c1-2060-4625-9911-452b33744ea8_characterizeImage_945756_44.stderr parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/44b7a6eb-9e02-4470-b723-f80d538ca7c0_isr_945756_44.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/44b7a6eb-9e02-4470-b723-f80d538ca7c0_isr_945756_44.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73771417257278fdf4339d305913df6e with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash b1c9a4e12206d76c08134ddc7ddd3d8b parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246402200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 942 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/7e5f4661-d697-4084-919d-4f2c2919ffb7_characterizeImage_945867_22.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/7e5f4661-d697-4084-919d-4f2c2919ffb7_characterizeImage_945867_22.stderr parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/690a7304-6757-4d5e-9071-af237c446826_isr_945867_22.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945867/690a7304-6757-4d5e-9071-af237c446826_isr_945867_22.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85026505e44db6cb41e0d731b5a8195b with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 8ba904eaecc5e2aa7e45218dbe4e8497 parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246402f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 943 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/86c57a26-7a15-46ed-8874-e8b53f9d2328_characterizeImage_945755_10.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/86c57a26-7a15-46ed-8874-e8b53f9d2328_characterizeImage_945755_10.stderr parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/224885b0-1295-46d7-bd02-5c37d6344487_isr_945755_10.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/224885b0-1295-46d7-bd02-5c37d6344487_isr_945755_10.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry deb2537ea30c7a87fba47c708440f69f with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 6f44056c52c81499b9fa3a61cdbe3094 parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 944 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/13ee89be-7195-4ef4-8c17-418966cc4353_characterizeImage_945757_27.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/13ee89be-7195-4ef4-8c17-418966cc4353_characterizeImage_945757_27.stderr parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/eae70657-8f36-47f4-a892-9645af59539a_isr_945757_27.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/eae70657-8f36-47f4-a892-9645af59539a_isr_945757_27.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6a90a47eefbae86da3943951b5d63aa with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash cbc4f9f61657d04e5e504f8a2f20a1ec parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 945 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/392f5067-62b1-472f-8c0e-dd39bcec876e_characterizeImage_945757_37.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/392f5067-62b1-472f-8c0e-dd39bcec876e_characterizeImage_945757_37.stderr parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/46c14efe-5889-4c4a-b614-5e338201c125_isr_945757_37.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/46c14efe-5889-4c4a-b614-5e338201c125_isr_945757_37.stderr parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d323f95c2f350767bc0b78e13be60649 with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 53e2e108b8b32f90668894393f315e7a parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 946 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/ca976092-8cb6-4b69-8ec7-5909a5643943_characterizeImage_945869_1.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/ca976092-8cb6-4b69-8ec7-5909a5643943_characterizeImage_945869_1.stderr parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/3edfc5c7-ca73-40c4-9455-b6b76f231d45_isr_945869_1.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/3edfc5c7-ca73-40c4-9455-b6b76f231d45_isr_945869_1.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bd8858fffaced94cd23c45178192161 with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash d528b3dfb8820d88f2c6e197d170ea92 parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246403420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 947 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/64be75ee-8814-4d46-bdee-87bfd4d830ad_characterizeImage_945757_36.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/64be75ee-8814-4d46-bdee-87bfd4d830ad_characterizeImage_945757_36.stderr parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/823551f6-139c-42cf-acb5-8b1922ee8cb2_isr_945757_36.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/823551f6-139c-42cf-acb5-8b1922ee8cb2_isr_945757_36.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5f216b8336669b2341240a597f07e23 with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 38bcaa744d81997bb36dc5f0a3a4d858 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246402c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 948 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0c1cc7d0-9ad4-4ee2-aef1-292e0ce53797_characterizeImage_945868_44.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0c1cc7d0-9ad4-4ee2-aef1-292e0ce53797_characterizeImage_945868_44.stderr parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/6ec1231c-a89f-4c8e-b18c-6f8c536a5eba_isr_945868_44.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945868/6ec1231c-a89f-4c8e-b18c-6f8c536a5eba_isr_945868_44.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb3f4a22956121b1f3b0680b4139021c with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/787cac3f-a298-47f2-9b86-47dd451d529e_calibrate_945757_1.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/787cac3f-a298-47f2-9b86-47dd451d529e_calibrate_945757_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:08 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40bcbe886b31243b7cd736b75f4ef3e9 with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/af125865-e34a-4a76-a7e2-a9ebe50f464e_calibrate_945755_27.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/af125865-e34a-4a76-a7e2-a9ebe50f464e_calibrate_945755_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:08 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 1096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 528fee933666b9e80d16285f4e960cfd with result from task 1096 parsl.dataflow.dflow DEBUG: Task 1096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1097 has memoization hash cbbdf312bb5bfcf5c1915d3566edf82d parsl.dataflow.memoization INFO: Task 1097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1097 try 0 launched on executor multi with executor id 949 parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/5c0b3fd9-8e29-4b9a-b660-4cdb60c0ef84_calibrate_945867_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/5c0b3fd9-8e29-4b9a-b660-4cdb60c0ef84_calibrate_945867_3.stderr parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/cef9edf2-6788-410a-ab77-f0843b946e4c_characterizeImage_945867_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/cef9edf2-6788-410a-ab77-f0843b946e4c_characterizeImage_945867_3.stderr parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02a4b2f007dcd34e7c48717ace6d0698 with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 96215f0dc793b03c75ba98c360485b1d parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246310900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 950 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79aa65e0-6a2c-4c92-82c4-e1477a52a589_calibrate_945868_3.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79aa65e0-6a2c-4c92-82c4-e1477a52a589_calibrate_945868_3.stderr parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/767e3b0f-3ec9-487e-a2cd-46b019d38825_characterizeImage_945868_3.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/767e3b0f-3ec9-487e-a2cd-46b019d38825_characterizeImage_945868_3.stderr parsl.dataflow.dflow INFO: Task 1081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3561240f1ef26fc7349559f82423b45 with result from task 1081 parsl.dataflow.dflow DEBUG: Task 1081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1082 has memoization hash 0c95612bb0957c98d6aed3eef1efd656 parsl.dataflow.memoization INFO: Task 1082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1082 try 0 launched on executor multi with executor id 951 parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/5dbdb2c9-25fc-48c8-889d-da65352d0a79_calibrate_945868_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/5dbdb2c9-25fc-48c8-889d-da65352d0a79_calibrate_945868_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/4f641f87-7d29-4a14-b0bb-8ed823539c97_characterizeImage_945868_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/4f641f87-7d29-4a14-b0bb-8ed823539c97_characterizeImage_945868_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:09 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f25823ce4ef29eef545f936b9df35f5e with result from task 971 parsl.dataflow.dflow DEBUG: Task 971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/e8bb4d2c-360f-447d-a721-2446f0838189_calibrate_945756_17.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/e8bb4d2c-360f-447d-a721-2446f0838189_calibrate_945756_17.stderr parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95606d09081dbdeb4976eb73fe7bd9d7 with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash 4edcd07475f0b0a3e09b6aea6446b4d2 parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 952 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8d20cc05-cc93-48e7-aa06-8ae431cad7f0_calibrate_945756_12.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8d20cc05-cc93-48e7-aa06-8ae431cad7f0_calibrate_945756_12.stderr parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b5f91a35-5cc9-4507-aee2-30ae10146e58_characterizeImage_945756_12.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b5f91a35-5cc9-4507-aee2-30ae10146e58_characterizeImage_945756_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:09 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00d284721d55a4292200943ce077e699 with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 63677406e0c8968e68c7c980ac932158 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 953 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/cd88c30e-fa51-4375-8432-3591efba05e7_calibrate_945869_13.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/cd88c30e-fa51-4375-8432-3591efba05e7_calibrate_945869_13.stderr parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/a9a29306-aa2a-4a76-847d-f05602234e2c_characterizeImage_945869_13.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/a9a29306-aa2a-4a76-847d-f05602234e2c_characterizeImage_945869_13.stderr parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15f8bfc18aa2f1953fe63b61453fd46f with result from task 862 parsl.dataflow.dflow DEBUG: Task 862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 863 has memoization hash bdc793b7250c18b6f1e7439f5b4c9ea3 parsl.dataflow.memoization INFO: Task 863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aa480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 954 parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/ac9f7885-1e27-4437-b46b-7f29a203c4d1_calibrate_945867_28.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/ac9f7885-1e27-4437-b46b-7f29a203c4d1_calibrate_945867_28.stderr parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/a83343b1-6d18-453a-9832-2d877e00819b_characterizeImage_945867_28.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/a83343b1-6d18-453a-9832-2d877e00819b_characterizeImage_945867_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:10 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5543a28fb02c357605d9a3dbff59ab8 with result from task 772 parsl.dataflow.dflow DEBUG: Task 772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 773 has memoization hash 7ee65e2852b2bea720da4c89487e3a2c parsl.dataflow.memoization INFO: Task 773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 955 parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/67a21569-a691-4848-a243-54711cff21a7_calibrate_945868_22.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/67a21569-a691-4848-a243-54711cff21a7_calibrate_945868_22.stderr parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/3af11c3f-5b28-4ba4-898f-27f77d8e70af_characterizeImage_945868_22.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/3af11c3f-5b28-4ba4-898f-27f77d8e70af_characterizeImage_945868_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:10 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38fb5e5fe5a0cddbfd223bde616d4432 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/0bb6ba42-6c3b-4d3e-a112-04046b346a54_calibrate_945756_62.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/0bb6ba42-6c3b-4d3e-a112-04046b346a54_calibrate_945756_62.stderr parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0be21ad542cc6e0bf6ffbb8b23dc4d12 with result from task 817 parsl.dataflow.dflow DEBUG: Task 817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash aeb38d6db1fd7a16a96bb9771a9528a5 parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 956 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b9d1d390-934e-4c1e-9010-8a2e557e071e_calibrate_945868_31.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b9d1d390-934e-4c1e-9010-8a2e557e071e_calibrate_945868_31.stderr parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/91698a2d-4208-4544-af73-5615a7fc30dc_characterizeImage_945868_31.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/91698a2d-4208-4544-af73-5615a7fc30dc_characterizeImage_945868_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:10 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cbeaa09eb8cd7d2fbae862376ae7c2f with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 4bd7302910a5b22433e7515bb2797716 parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bdee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 957 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2e5d5288-0bf3-4eeb-8c3b-ae6ebbdbcd95_calibrate_945868_20.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2e5d5288-0bf3-4eeb-8c3b-ae6ebbdbcd95_calibrate_945868_20.stderr parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/e27ccb4e-4b6e-44fc-9467-fe4f9663bbb2_characterizeImage_945868_20.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/e27ccb4e-4b6e-44fc-9467-fe4f9663bbb2_characterizeImage_945868_20.stderr parsl.dataflow.dflow INFO: Task 802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2b835e876e85314109e8f43680c2ee3 with result from task 802 parsl.dataflow.dflow DEBUG: Task 802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 803 has memoization hash 62f9d9ec3f9e3494f9128443878b90fe parsl.dataflow.memoization INFO: Task 803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463465c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 958 parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/5d8fe282-7bab-4c32-85f1-6217ee2d4105_calibrate_945757_47.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/5d8fe282-7bab-4c32-85f1-6217ee2d4105_calibrate_945757_47.stderr parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a95e41b9-93fb-4718-9dd3-370b390cf9bc_characterizeImage_945757_47.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a95e41b9-93fb-4718-9dd3-370b390cf9bc_characterizeImage_945757_47.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ff8a12b791a551835e95aec37b8ad4d with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/da0bd654-daaa-4923-aab5-c9fc713c42a2_calibrate_945757_45.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/da0bd654-daaa-4923-aab5-c9fc713c42a2_calibrate_945757_45.stderr parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3235f85325d58cfd9e574092b0e2891 with result from task 770 parsl.dataflow.dflow DEBUG: Task 770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/a7d1b8d9-c6fb-41ab-bd92-9539c1e61b00_calibrate_945756_42.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/a7d1b8d9-c6fb-41ab-bd92-9539c1e61b00_calibrate_945756_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:11 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b01dabfc20f75f8cd8df39f7736f3f8 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 21d77a6a61bf4e319d475a8cc23c5269 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 959 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/391b3755-316b-41fc-bd39-582cc3f06cda_calibrate_945867_38.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/391b3755-316b-41fc-bd39-582cc3f06cda_calibrate_945867_38.stderr parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/1bb69e89-133f-408d-93bb-5f932a70dd90_characterizeImage_945867_38.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/1bb69e89-133f-408d-93bb-5f932a70dd90_characterizeImage_945867_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:11 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4791abebe38f99b596afa547d67fe97 with result from task 941 parsl.dataflow.dflow DEBUG: Task 941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/a3419303-94c3-4f56-82dd-6f156f2c4d2b_calibrate_945757_58.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/a3419303-94c3-4f56-82dd-6f156f2c4d2b_calibrate_945757_58.stderr parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acd3ccfc783d1b77f5e2516edee24e4c with result from task 779 parsl.dataflow.dflow DEBUG: Task 779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/064e3f3e-12f1-454c-a18b-f82e90c6bfcb_calibrate_945755_43.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/064e3f3e-12f1-454c-a18b-f82e90c6bfcb_calibrate_945755_43.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de8cffe558c7ba67a3bb4174c50df439 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/99a69e7f-ba6b-4fad-8996-427c269854ab_calibrate_945757_43.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/99a69e7f-ba6b-4fad-8996-427c269854ab_calibrate_945757_43.stderr parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d4211db9d41185c8709b2b6b08007ef with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash 7deb2c315eab168f540595927b621fb6 parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 960 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/e61c675d-82c2-4598-b823-618d1d6e5bcc_calibrate_945869_32.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/e61c675d-82c2-4598-b823-618d1d6e5bcc_calibrate_945869_32.stderr parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/621c1adf-bf2f-4676-83de-a3ddcccee9b2_characterizeImage_945869_32.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/621c1adf-bf2f-4676-83de-a3ddcccee9b2_characterizeImage_945869_32.stderr parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aff0bc1a4011f1c01f935d1c59bb8981 with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash 60d32aafe849ae04e8e04f031b0d48da parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 961 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d5318749-75a6-4c1e-95ad-02dc1ea33a5a_calibrate_945867_39.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d5318749-75a6-4c1e-95ad-02dc1ea33a5a_calibrate_945867_39.stderr parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/868de034-4bc3-41a0-8d14-98a1a6c6555e_characterizeImage_945867_39.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/868de034-4bc3-41a0-8d14-98a1a6c6555e_characterizeImage_945867_39.stderr parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4000c658733fdc6633914a13b5dfbc62 with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f2723b48-d5fe-4264-b42d-27976316ae85_calibrate_945755_9.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f2723b48-d5fe-4264-b42d-27976316ae85_calibrate_945755_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:12 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13285ea500519c1cc23ef5a1b32ceff5 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/2b4632d2-991d-4f5c-8d0f-7d583668e2ee_calibrate_945755_32.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/2b4632d2-991d-4f5c-8d0f-7d583668e2ee_calibrate_945755_32.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 024e5509c664d48fe302a8481ebf1266 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/48c07312-8790-4dae-bbbc-6ae0db432964_calibrate_945867_27.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/48c07312-8790-4dae-bbbc-6ae0db432964_calibrate_945867_27.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2067f270835d5d77855c5dae4740650 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/ac012908-d11a-4e4a-bd64-1751627fa4c1_calibrate_945757_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/ac012908-d11a-4e4a-bd64-1751627fa4c1_calibrate_945757_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:12 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acfa9bb00fc3c53ac5ea4cdf14007903 with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/15f814c7-c3dc-49ef-8982-c37e7b31142a_calibrate_945869_33.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/15f814c7-c3dc-49ef-8982-c37e7b31142a_calibrate_945869_33.stderr parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f56c04452dbb40259972d155f9f6d842 with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash b3733c36746d057a112c041da3dc61ba parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246400fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 962 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/2c49a2ce-729b-4e1d-8c1b-46795ce508bd_calibrate_945869_40.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/2c49a2ce-729b-4e1d-8c1b-46795ce508bd_calibrate_945869_40.stderr parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/7daa1979-0ee7-4edd-960c-ac19b18b1d5b_characterizeImage_945869_40.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/7daa1979-0ee7-4edd-960c-ac19b18b1d5b_characterizeImage_945869_40.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92b131f870e5ed1933413d51fbbbe8c3 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash adaefc9ea30373fb2e4ff617d2539ee9 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247915940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 963 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/69de406a-2cfe-454d-a524-98bd0f3cebde_calibrate_945869_5.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/69de406a-2cfe-454d-a524-98bd0f3cebde_calibrate_945869_5.stderr parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/e832b6f3-739d-43e8-98dd-aacb42b6d297_characterizeImage_945869_5.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/e832b6f3-739d-43e8-98dd-aacb42b6d297_characterizeImage_945869_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:13 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8576e548ea099f9ebddf1c190f3a2c75 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/7037cf13-3e2c-4cfe-ac61-d71cc1424ae0_calibrate_945869_46.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/7037cf13-3e2c-4cfe-ac61-d71cc1424ae0_calibrate_945869_46.stderr parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc444c226ef44591feab997f085b2b06 with result from task 776 parsl.dataflow.dflow DEBUG: Task 776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c3e46a8d-74cd-4e4c-9948-4a9ac6332350_calibrate_945756_21.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c3e46a8d-74cd-4e4c-9948-4a9ac6332350_calibrate_945756_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:13 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:13 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0eb539d49f52d49c57371f357efa29c7 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/1d43f2a4-eaf5-4df6-89a4-a0979bc08891_calibrate_945867_5.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/1d43f2a4-eaf5-4df6-89a4-a0979bc08891_calibrate_945867_5.stderr parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f992fd89410464843402dd437ba4895 with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/8e7cb3f7-9a3e-46c5-a180-7fde06f5dc98_calibrate_945755_60.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/8e7cb3f7-9a3e-46c5-a180-7fde06f5dc98_calibrate_945755_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:14 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a51c84b146d597375c2eeee0e76f234 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/d6eed2e3-605a-4f06-9940-24c0557a5be5_calibrate_945868_50.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/d6eed2e3-605a-4f06-9940-24c0557a5be5_calibrate_945868_50.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38521d12c90f5ec9fc7da7d9083cb34b with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/76aa4329-8ff3-44aa-9674-d4d79974b46f_calibrate_945869_48.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/76aa4329-8ff3-44aa-9674-d4d79974b46f_calibrate_945869_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:15 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d063f982118832ad3ae305888e2b4665 with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash 0b638c846a170a65cb348ff53b1f8fd4 parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462840e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 964 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/61d235fc-a98f-4161-9bd1-20bdc19321c1_calibrate_945755_49.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/61d235fc-a98f-4161-9bd1-20bdc19321c1_calibrate_945755_49.stderr parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/f0e2a9a6-bfd2-4685-9c37-cf38baa51fc8_characterizeImage_945755_49.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/f0e2a9a6-bfd2-4685-9c37-cf38baa51fc8_characterizeImage_945755_49.stderr parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fb91040281baf89cdf2e276dc1b7b1a with result from task 805 parsl.dataflow.dflow DEBUG: Task 805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash 610093f6f66164768b434cf546c2d9b6 parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463468e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 965 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/aa73124c-9116-48a2-8ae0-d2a7bcafe64e_calibrate_945755_50.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/aa73124c-9116-48a2-8ae0-d2a7bcafe64e_calibrate_945755_50.stderr parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/fb159200-ba8d-426f-9652-dd160855f374_characterizeImage_945755_50.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/fb159200-ba8d-426f-9652-dd160855f374_characterizeImage_945755_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:15 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 911882864613e4db4f7f3ea023b817a1 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/a6bb827d-c7b6-4c35-8fdf-912d92bcf878_calibrate_945869_53.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/a6bb827d-c7b6-4c35-8fdf-912d92bcf878_calibrate_945869_53.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 187dc581a90ea3a38ef5d6e367618f13 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 962acdb341ab5d825bd182abea32fd38 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247917ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 966 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8cfc81f1-8e39-49da-82f1-f45c19acdac4_calibrate_945868_29.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8cfc81f1-8e39-49da-82f1-f45c19acdac4_calibrate_945868_29.stderr parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a19cacf9-e8fc-4b26-b8d0-7d27e1d34a94_characterizeImage_945868_29.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a19cacf9-e8fc-4b26-b8d0-7d27e1d34a94_characterizeImage_945868_29.stderr parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 052fb6949a727dfb3112fd7ee0df082f with result from task 755 parsl.dataflow.dflow DEBUG: Task 755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/6c090abd-d51d-4aae-8d5f-1135456a8f6a_calibrate_945756_37.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/6c090abd-d51d-4aae-8d5f-1135456a8f6a_calibrate_945756_37.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1126d732a4e13d245aa5ee9aeefabdc6 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/59aaa762-6f8b-4b80-ad19-99050ba2ffa3_calibrate_945869_27.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/59aaa762-6f8b-4b80-ad19-99050ba2ffa3_calibrate_945869_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:16 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acc19be14496fb63eb02665da66d0745 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2fbf43fd-26c3-447c-917c-aed553a6f9e8_calibrate_945868_27.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2fbf43fd-26c3-447c-917c-aed553a6f9e8_calibrate_945868_27.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b8a952a5a3f483cd741023bb8fdf4f1 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash fe84007f2565a53beedcec196d35516e parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463928e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 967 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/d82f216d-14de-4cad-b130-83f8223bd35f_calibrate_945868_17.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/d82f216d-14de-4cad-b130-83f8223bd35f_calibrate_945868_17.stderr parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/1d7b610d-3401-43de-9362-91339162d783_characterizeImage_945868_17.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/1d7b610d-3401-43de-9362-91339162d783_characterizeImage_945868_17.stderr parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3551c6671182b3b0b8b7034c8356d739 with result from task 854 parsl.dataflow.dflow DEBUG: Task 854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8729d606-33e5-4ba6-8547-aa3cdf6224b5_calibrate_945756_13.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8729d606-33e5-4ba6-8547-aa3cdf6224b5_calibrate_945756_13.stderr parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c62b81ca879d55e225b3379b6dd3374 with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d2c88b85-0c66-409c-9c47-1421641012b5_calibrate_945867_13.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d2c88b85-0c66-409c-9c47-1421641012b5_calibrate_945867_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:16 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf1687801b965460057ad36c46709ef1 with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/3541ef42-41a7-46a7-bcb8-4e117ef62f0f_calibrate_945867_46.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/3541ef42-41a7-46a7-bcb8-4e117ef62f0f_calibrate_945867_46.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dd3b8deed40204d58cd497aad30d350 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/73d7fe3a-3b95-4d7b-8ab8-dd0bb9f5eb17_calibrate_945756_50.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/73d7fe3a-3b95-4d7b-8ab8-dd0bb9f5eb17_calibrate_945756_50.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdf0a8ba71d82026282ef5046f997166 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/7de76ff1-d91a-4dab-8dfa-4f6e56799f92_calibrate_945867_37.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/7de76ff1-d91a-4dab-8dfa-4f6e56799f92_calibrate_945867_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:17 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70bb369dbf4b7aa6d3e92d2f4c788af2 with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash d4c09f953c5e11695b53ba3ec73a6054 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 968 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d4d6e0b3-e314-41e6-a7c3-84e79d043b2f_calibrate_945867_31.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d4d6e0b3-e314-41e6-a7c3-84e79d043b2f_calibrate_945867_31.stderr parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/b5b60afa-c541-4296-90c4-17eaf426ae32_characterizeImage_945867_31.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/b5b60afa-c541-4296-90c4-17eaf426ae32_characterizeImage_945867_31.stderr parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79308fbab5ec65825fc39130fd3ca90a with result from task 800 parsl.dataflow.dflow DEBUG: Task 800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/678dd4aa-5b29-4461-a57c-4dd9c36f8086_calibrate_945755_17.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/678dd4aa-5b29-4461-a57c-4dd9c36f8086_calibrate_945755_17.stderr parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 450822c155d39f09d3913e986d630f5a with result from task 848 parsl.dataflow.dflow DEBUG: Task 848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b0d5ae4d-c5a3-4586-b234-57987e768b7f_calibrate_945869_47.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b0d5ae4d-c5a3-4586-b234-57987e768b7f_calibrate_945869_47.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7548b14687789ac10e31fcdf8f2ae10c with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 677e3dd4bc2b0787aa6bce8301d66b9f parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 969 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/76b00b96-b287-4462-9337-ecc13adee591_calibrate_945756_40.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/76b00b96-b287-4462-9337-ecc13adee591_calibrate_945756_40.stderr parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/3cdbc5df-88d4-49dd-97d5-6a59acc2cb09_characterizeImage_945756_40.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/3cdbc5df-88d4-49dd-97d5-6a59acc2cb09_characterizeImage_945756_40.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8b14fc67e82fa958f4a357535109438 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/140b045d-d01d-4c40-ba4d-945d886754be_calibrate_945757_48.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/140b045d-d01d-4c40-ba4d-945d886754be_calibrate_945757_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:17 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 133d575763184a10bd54fe6db90180da with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/17ba65ad-a7fb-4610-9e8d-00d2253fde13_calibrate_945868_58.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/17ba65ad-a7fb-4610-9e8d-00d2253fde13_calibrate_945868_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:18 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 860af77e5084ebaa4b11f99818feaff3 with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/3afa57bb-fc28-4fb0-a3db-22adebf9a9cf_calibrate_945867_20.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/3afa57bb-fc28-4fb0-a3db-22adebf9a9cf_calibrate_945867_20.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95e976bc5a08298a26dab4d09c5e34f7 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/e030b1d7-6014-4c6b-bf3d-af0efb83e2aa_calibrate_945869_51.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/e030b1d7-6014-4c6b-bf3d-af0efb83e2aa_calibrate_945869_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:18 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ad06d7ccb8b8ba97259a2a6a07bbbcb with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/97bf7bb3-bc6b-442c-abf1-f6364953eab2_calibrate_945755_34.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/97bf7bb3-bc6b-442c-abf1-f6364953eab2_calibrate_945755_34.stderr parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28010ef00dbc55a1eec319b8eb0efdbd with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash c428652d6267f44f3cf1952d33dd5f04 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 970 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/63b6da71-72af-47be-9e5f-5df6c0492fef_calibrate_945867_60.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/63b6da71-72af-47be-9e5f-5df6c0492fef_calibrate_945867_60.stderr parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/89b844e5-3d1d-4075-b19c-ab6fe2603061_characterizeImage_945867_60.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/89b844e5-3d1d-4075-b19c-ab6fe2603061_characterizeImage_945867_60.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f769e81fd57c69afa4e86f49b5f346a with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/db8e32fc-643a-4541-9943-a6c418156c43_calibrate_945869_60.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/db8e32fc-643a-4541-9943-a6c418156c43_calibrate_945869_60.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec7e63387329bce623fd3eca4da4af0d with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/6912b654-537f-4bf7-9659-bc957f6ab1c8_calibrate_945867_40.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/6912b654-537f-4bf7-9659-bc957f6ab1c8_calibrate_945867_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:19 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8496381540c1d3d5a10a894ec06ce0d7 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/093a1fdd-5da7-4f3d-9a91-ad2118430097_calibrate_945869_56.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/093a1fdd-5da7-4f3d-9a91-ad2118430097_calibrate_945869_56.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33abfec5e5818a90ffb93ff4baabae68 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/902b7813-7713-45dc-8cd8-f0675abc093f_calibrate_945868_37.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/902b7813-7713-45dc-8cd8-f0675abc093f_calibrate_945868_37.stderr parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1f10264a14a7cc125b5ccda685066dd with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/39d5929c-f297-458b-9639-babc45f9378e_calibrate_945867_11.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/39d5929c-f297-458b-9639-babc45f9378e_calibrate_945867_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:20 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4af508c2e047727aebb03654032afaf with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/10fdab73-3c13-4646-98c9-5eae55933037_calibrate_945869_34.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/10fdab73-3c13-4646-98c9-5eae55933037_calibrate_945869_34.stderr parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b94d75c73cf34594405115048141f32 with result from task 835 parsl.dataflow.dflow DEBUG: Task 835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 836 has memoization hash 79fa7bd4e7abac569d961309beb00dc3 parsl.dataflow.memoization INFO: Task 836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 971 parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/bce82a5f-c68f-4361-a9c7-9d30aa13b4fd_calibrate_945867_62.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/bce82a5f-c68f-4361-a9c7-9d30aa13b4fd_calibrate_945867_62.stderr parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/8a1cd2a5-b287-41af-bb27-577a4e47454d_characterizeImage_945867_62.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/8a1cd2a5-b287-41af-bb27-577a4e47454d_characterizeImage_945867_62.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaedaaaac09ed666da62a9668d5c24d1 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/db7480ef-3d40-4420-bf8e-6c2612e2653f_calibrate_945757_7.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/db7480ef-3d40-4420-bf8e-6c2612e2653f_calibrate_945757_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:20 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31e1cf9742012f3f089ddcaa8379f8da with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash f9a4e30b97e188e6289bb32e00d3ea2e parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 972 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/52ed28fc-7c3d-49fe-8afc-8b7611f27df0_calibrate_945869_15.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/52ed28fc-7c3d-49fe-8afc-8b7611f27df0_calibrate_945869_15.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/881a4e45-8dd0-4d15-9d99-ec7bb24c3a25_characterizeImage_945869_15.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/881a4e45-8dd0-4d15-9d99-ec7bb24c3a25_characterizeImage_945869_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:21 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07e5b1eb7eb840d92210a9db5dbf5c31 with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d1e71f44-6085-4ff6-ba0d-e99a73eb93a6_calibrate_945755_3.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d1e71f44-6085-4ff6-ba0d-e99a73eb93a6_calibrate_945755_3.stderr parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dd13a42cdf4520a65b12ee238e80ad3 with result from task 895 parsl.dataflow.dflow DEBUG: Task 895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 896 has memoization hash abc11387f4b851174bd4ce6c121ab5ae parsl.dataflow.memoization INFO: Task 896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 973 parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/e4599609-6468-419d-aaed-5f40c5ee4015_calibrate_945756_31.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/e4599609-6468-419d-aaed-5f40c5ee4015_calibrate_945756_31.stderr parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f50cbe57-8e97-4df4-81bf-7b32e36fe016_characterizeImage_945756_31.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f50cbe57-8e97-4df4-81bf-7b32e36fe016_characterizeImage_945756_31.stderr parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1491f8bb21d6fb5a154522303538f36 with result from task 905 parsl.dataflow.dflow DEBUG: Task 905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/e294ed5b-a034-408e-b6e5-b5e46f6851ee_calibrate_945757_12.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/e294ed5b-a034-408e-b6e5-b5e46f6851ee_calibrate_945757_12.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78a968adbf7cb06419e3e404715cf1a9 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/40bab741-f770-407f-b5c4-8748c2ebe95b_calibrate_945869_10.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/40bab741-f770-407f-b5c4-8748c2ebe95b_calibrate_945869_10.stderr parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 092ca735b775d91093526df3b32b2ed1 with result from task 868 parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:21 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow DEBUG: Task 868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 869 has memoization hash a40d57de6920d5eb3b038113b02bbb14 parsl.dataflow.memoization INFO: Task 869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 974 parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1a91f50a-7cf4-4ee2-8d84-d2d79cbb9e3a_calibrate_945869_19.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1a91f50a-7cf4-4ee2-8d84-d2d79cbb9e3a_calibrate_945869_19.stderr parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/cb88571a-3a1e-4ffa-b27b-53b1a0f0978e_characterizeImage_945869_19.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/cb88571a-3a1e-4ffa-b27b-53b1a0f0978e_characterizeImage_945869_19.stderr parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 436b1be034659bf7661ae7c173301ec6 with result from task 886 parsl.dataflow.dflow DEBUG: Task 886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 887 has memoization hash 976c97bce665253cae47b68690bebaae parsl.dataflow.memoization INFO: Task 887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461abd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 975 parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/0dbc9925-dc4c-48fe-9081-a34a253241f9_calibrate_945757_23.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/0dbc9925-dc4c-48fe-9081-a34a253241f9_calibrate_945757_23.stderr parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/4f26ac23-2d58-476a-8d20-f572cd747cad_characterizeImage_945757_23.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/4f26ac23-2d58-476a-8d20-f572cd747cad_characterizeImage_945757_23.stderr parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a32ae9431a4c8ca046ed0ea58e7db81 with result from task 824 parsl.dataflow.dflow DEBUG: Task 824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/6290a584-d78c-4d3b-91a8-4750a30a2912_calibrate_945868_13.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/6290a584-d78c-4d3b-91a8-4750a30a2912_calibrate_945868_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:22 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 664bc35da0953bb55916ac0f66967e69 with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash 065bfe60221d6665a68336b42b2d788a parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462876a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 976 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d2923cbf-b1a9-4dc1-8068-f14d10975357_calibrate_945867_49.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d2923cbf-b1a9-4dc1-8068-f14d10975357_calibrate_945867_49.stderr parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/31238fe6-3138-499b-aa4a-2a1c2811b2a4_characterizeImage_945867_49.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/31238fe6-3138-499b-aa4a-2a1c2811b2a4_characterizeImage_945867_49.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fa24061c98877392abb6fd68790ea1b with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 8ccc05a6be84028ed87e817dc8866a98 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 977 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/964243a1-3945-426a-a763-2d80eafe2027_calibrate_945867_29.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/964243a1-3945-426a-a763-2d80eafe2027_calibrate_945867_29.stderr parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ede4b03b-b547-415d-a8ed-f2b96aaa54b6_characterizeImage_945867_29.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ede4b03b-b547-415d-a8ed-f2b96aaa54b6_characterizeImage_945867_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:22 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4adcd9e3a3f9e3c3f8a51a62ab9345f with result from task 883 parsl.dataflow.dflow DEBUG: Task 883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 884 has memoization hash 135758735141bfe932429ed57649751b parsl.dataflow.memoization INFO: Task 884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461aba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 978 parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/1c1efa15-b7dd-41d7-9d4b-7b74ed23700c_calibrate_945868_41.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/1c1efa15-b7dd-41d7-9d4b-7b74ed23700c_calibrate_945868_41.stderr parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/9c91aef9-a5ed-42a2-96d2-ef9c4a5bf32f_characterizeImage_945868_41.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/9c91aef9-a5ed-42a2-96d2-ef9c4a5bf32f_characterizeImage_945868_41.stderr parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cca47ea38f34b056e24a7eb412051b33 with result from task 845 parsl.dataflow.dflow DEBUG: Task 845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/da524889-dab4-487e-855c-215be3efd9aa_calibrate_945756_49.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/da524889-dab4-487e-855c-215be3efd9aa_calibrate_945756_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:23 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36f157d18b81f633ea5c9f9bc3476ee6 with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 7678c87b7162b97e9c240e26f01ad655 parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 979 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b5767f11-9522-4a31-b1c8-a5767bea913e_calibrate_945868_6.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b5767f11-9522-4a31-b1c8-a5767bea913e_calibrate_945868_6.stderr parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/ddaa29be-f357-4a7d-91a7-ad9db322191f_characterizeImage_945868_6.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/ddaa29be-f357-4a7d-91a7-ad9db322191f_characterizeImage_945868_6.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0aa2016817e3d61087955d34bcf9f425 with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash 13ce0e34e9c50adb90a0d5636392ca87 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 980 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/1a545bee-a5fe-4382-9d79-b63522e70f2c_calibrate_945868_8.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/1a545bee-a5fe-4382-9d79-b63522e70f2c_calibrate_945868_8.stderr parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/2ff24112-6dcf-42a6-a3f5-71155d5ab7f3_characterizeImage_945868_8.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/2ff24112-6dcf-42a6-a3f5-71155d5ab7f3_characterizeImage_945868_8.stderr parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05388d66d1348138818d5c5a994e6b8a with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/3b06709d-317e-41f0-8eda-12105b4ee6b6_calibrate_945755_53.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/3b06709d-317e-41f0-8eda-12105b4ee6b6_calibrate_945755_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:24 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aec8578e6892ad2543edbc3386397059 with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash af9d02360ab88f1bdef4c5b0d7912350 parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 981 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/9a352527-3ac2-4dc8-ba3b-27b2098170da_calibrate_945867_10.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/9a352527-3ac2-4dc8-ba3b-27b2098170da_calibrate_945867_10.stderr parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/7f14d69f-accb-4be7-a3bd-cca87839e1a3_characterizeImage_945867_10.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/7f14d69f-accb-4be7-a3bd-cca87839e1a3_characterizeImage_945867_10.stderr parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4c82cba98e1ac8ca1acb029355d8474 with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash 7c3bc5109c6484adca0efb8f44910974 parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462bcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 982 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/693989fd-a547-416f-9677-27efeef008e8_calibrate_945757_19.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/693989fd-a547-416f-9677-27efeef008e8_calibrate_945757_19.stderr parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a6e43120-948b-4fba-9cbc-fdd58d567c5c_characterizeImage_945757_19.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a6e43120-948b-4fba-9cbc-fdd58d567c5c_characterizeImage_945757_19.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b74b9ebd657a173316226ebe68b8d290 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/bbd84d5a-9805-431d-b28b-75774f91659a_calibrate_945756_9.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/bbd84d5a-9805-431d-b28b-75774f91659a_calibrate_945756_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:24 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76148e1675a083d9dc287c4f3d8aaa63 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash f167d103250465c67334955dc47dbf0c parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924794f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 983 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/55408845-e6d5-4b76-b2d9-981be02dd657_calibrate_945867_30.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/55408845-e6d5-4b76-b2d9-981be02dd657_calibrate_945867_30.stderr parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e0f148f9-ade4-42d3-8b16-418e4dd16d27_characterizeImage_945867_30.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e0f148f9-ade4-42d3-8b16-418e4dd16d27_characterizeImage_945867_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:25 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e3bdb6917bd7738aa514c0a6d362728 with result from task 898 parsl.dataflow.dflow DEBUG: Task 898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 899 has memoization hash 9d19d50962bf2b1f77aaf198a6c3eff7 parsl.dataflow.memoization INFO: Task 899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 984 parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/4269cdcf-1ff1-4ae8-8384-83257574a98d_calibrate_945869_23.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/4269cdcf-1ff1-4ae8-8384-83257574a98d_calibrate_945869_23.stderr parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/9ba139c5-6e74-443c-b1aa-f83471c9dfc6_characterizeImage_945869_23.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/9ba139c5-6e74-443c-b1aa-f83471c9dfc6_characterizeImage_945869_23.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e2235b950546ddb754ec5b3f774e0c0 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/cd51cab8-1b4f-4af8-8ef4-5efecec640d3_calibrate_945869_9.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/cd51cab8-1b4f-4af8-8ef4-5efecec640d3_calibrate_945869_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:25 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41ab9ee151428c7254623837c200ae1b with result from task 850 parsl.dataflow.dflow DEBUG: Task 850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 851 has memoization hash 8531d9692a4349e1d6ecee51649c51bb parsl.dataflow.memoization INFO: Task 851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 985 parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0922e95e-3f5c-4520-bef7-16b6c16eede8_calibrate_945869_58.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0922e95e-3f5c-4520-bef7-16b6c16eede8_calibrate_945869_58.stderr parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2d68eb6e-18ce-4d92-88dc-cf73e7266c33_characterizeImage_945869_58.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2d68eb6e-18ce-4d92-88dc-cf73e7266c33_characterizeImage_945869_58.stderr parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53205458b27a23665e62929cc1eb0c74 with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash b0fb53ebdb45cef709571cc4ff6b10d2 parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462beb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 986 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/d46f222c-4de1-4608-a13b-c75f005e7257_calibrate_945868_62.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/d46f222c-4de1-4608-a13b-c75f005e7257_calibrate_945868_62.stderr parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/cfb4d370-eebb-4383-8b10-05f7b47cf5ac_characterizeImage_945868_62.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/cfb4d370-eebb-4383-8b10-05f7b47cf5ac_characterizeImage_945868_62.stderr parsl.dataflow.dflow INFO: Task 1085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry debdaaa865624c75d5877fbc56747001 with result from task 1085 parsl.dataflow.dflow DEBUG: Task 1085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/16dbe4cc-05d1-4bd9-805f-ba74f7b7d3f1_calibrate_945757_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/16dbe4cc-05d1-4bd9-805f-ba74f7b7d3f1_calibrate_945757_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:26 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06808e46c9ffe7226cd371adf483ea6f with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/659bd90d-a59d-473f-88e7-de0f2f76e0a0_calibrate_945869_41.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/659bd90d-a59d-473f-88e7-de0f2f76e0a0_calibrate_945869_41.stderr parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76395d6ff0085a9e69b0263ffa399467 with result from task 815 parsl.dataflow.dflow DEBUG: Task 815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/3d8e0d27-890b-4470-ae87-a041b21cd005_calibrate_945756_39.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/3d8e0d27-890b-4470-ae87-a041b21cd005_calibrate_945756_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:26 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdb9a36c7e372ca4c80e279d569d1d22 with result from task 1100 parsl.dataflow.dflow DEBUG: Task 1100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/b37118f2-5e9c-47eb-901c-2e2fdfedb6c4_calibrate_945757_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/b37118f2-5e9c-47eb-901c-2e2fdfedb6c4_calibrate_945757_53.stderr parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bcbc766d20e20d5e4151c58a7bfb64f with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/b5a8dacd-c87c-4d06-922e-93a4895652e2_calibrate_945867_58.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/b5a8dacd-c87c-4d06-922e-93a4895652e2_calibrate_945867_58.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7a64722431f58b191a4205e10202aca with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 98d16abeee8c47c3f57a24c886ee8d25 parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246390220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 987 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/fa011f88-9499-4d18-a637-2cb6ca333262_calibrate_945755_41.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/fa011f88-9499-4d18-a637-2cb6ca333262_calibrate_945755_41.stderr parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/9cd307bc-e425-4510-8c06-01524416f8cd_characterizeImage_945755_41.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/9cd307bc-e425-4510-8c06-01524416f8cd_characterizeImage_945755_41.stderr parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9086faebb450885ba1d3c50b3bd23a6b with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d3da880c-83a6-4aa3-8f63-7f3f830a8d53_calibrate_945757_31.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d3da880c-83a6-4aa3-8f63-7f3f830a8d53_calibrate_945757_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:27 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d2f0a14c0d2f442de2989829bfb0dcf with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/1149017f-6a2d-4a54-ab76-f19c3f4ad7a3_calibrate_945867_36.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/1149017f-6a2d-4a54-ab76-f19c3f4ad7a3_calibrate_945867_36.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f3b88128843ff80d85b1582fa80cf15 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/6e82457c-4a42-4302-a077-1425c7357d5a_calibrate_945757_20.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/6e82457c-4a42-4302-a077-1425c7357d5a_calibrate_945757_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:28 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc3547ced378fd98447a14f1ca99da2a with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 28af6fe178a6a17d846e584d907fee89 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479177e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 988 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f91db016-e9b0-41b4-b7bd-da8a5a08293e_calibrate_945867_7.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f91db016-e9b0-41b4-b7bd-da8a5a08293e_calibrate_945867_7.stderr parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/cd8dc4f4-e3b7-4421-8b4b-79a808b9e9f0_characterizeImage_945867_7.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/cd8dc4f4-e3b7-4421-8b4b-79a808b9e9f0_characterizeImage_945867_7.stderr parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19ed348f40b801a8127c5fc2c56c32b7 with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/6e9a0c9d-a0a2-4885-8a84-2a2022c961a0_calibrate_945868_52.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/6e9a0c9d-a0a2-4885-8a84-2a2022c961a0_calibrate_945868_52.stderr parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd6f7289c151b0b8efd928a5c0ee50f8 with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/d8976979-1c96-4ef4-a8dc-6eb4f6146e70_calibrate_945756_18.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/d8976979-1c96-4ef4-a8dc-6eb4f6146e70_calibrate_945756_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:28 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c7cdc69c1e6371961261404429bd086 with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/dba511ff-4b87-43b2-8f83-532ca7446aa3_calibrate_945757_9.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/dba511ff-4b87-43b2-8f83-532ca7446aa3_calibrate_945757_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:29 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99a3f241ac90ad6d73782883b75547e1 with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/93f61585-0b79-4725-845c-2be8dc40188c_calibrate_945868_53.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/93f61585-0b79-4725-845c-2be8dc40188c_calibrate_945868_53.stderr parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 370e6fda0f044335c244c8fa6a6a7aa8 with result from task 680 parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:29 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/c045ced5-96b3-4049-8da7-fb0795abbbed_calibrate_945868_40.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/c045ced5-96b3-4049-8da7-fb0795abbbed_calibrate_945868_40.stderr parsl.dataflow.dflow INFO: Task 956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36cd0cfafc54676d9cb9254dd9c0d961 with result from task 956 parsl.dataflow.dflow DEBUG: Task 956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/52c38800-3593-4344-94c6-94ad3e91577e_calibrate_945755_54.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/52c38800-3593-4344-94c6-94ad3e91577e_calibrate_945755_54.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49d8beff6b60c54468bed653d3549507 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ee638eaf-408c-48a0-b011-d9224d65c4da_calibrate_945868_24.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ee638eaf-408c-48a0-b011-d9224d65c4da_calibrate_945868_24.stderr parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dea282f8053e32353ca1b4c4de56bc19 with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d6e929af-17b5-4887-ba7e-d19f714bca29_calibrate_945755_29.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d6e929af-17b5-4887-ba7e-d19f714bca29_calibrate_945755_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:30 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 119e6968764fe08f4355a1b77fee9330 with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/be984a3a-961e-4bad-9ebb-25f43abf0580_calibrate_945867_8.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/be984a3a-961e-4bad-9ebb-25f43abf0580_calibrate_945867_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:30 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af4406501d4c2a69dbd874b541b2ad83 with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/9df2b344-dde2-4632-82e2-571d3334835b_calibrate_945868_51.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/9df2b344-dde2-4632-82e2-571d3334835b_calibrate_945868_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:31 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60acb854ccc46918bd076b1b5f6a582f with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/aa2a72a8-53f8-4385-a5fe-fc518613ae05_calibrate_945756_15.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/aa2a72a8-53f8-4385-a5fe-fc518613ae05_calibrate_945756_15.stderr parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de07a674c02bf436515a27e70b1980ff with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash 647ba9b12924510a65d93acd31ed651a parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 989 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/f79b6636-81df-4ba7-ae84-19082a3ce231_calibrate_945756_29.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/f79b6636-81df-4ba7-ae84-19082a3ce231_calibrate_945756_29.stderr parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b6ad3ef9-5561-4030-addb-8ebb2e901e8e_characterizeImage_945756_29.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b6ad3ef9-5561-4030-addb-8ebb2e901e8e_characterizeImage_945756_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:31 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1392c9abbbf74d9fcd80fd55f10ae36d with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/15b627c3-7203-4dbe-92da-1abd9a803bab_calibrate_945756_19.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/15b627c3-7203-4dbe-92da-1abd9a803bab_calibrate_945756_19.stderr parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce94fe54b741fabcf12a2e3fb940ceb7 with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 428858b8c64a928f1a238e8cc3004a86 parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 990 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c759e4a0-ead1-4dc8-b157-74725a07e8e6_calibrate_945867_33.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c759e4a0-ead1-4dc8-b157-74725a07e8e6_calibrate_945867_33.stderr parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/0e2987d5-26a8-4816-bbf1-86e74daf5871_characterizeImage_945867_33.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/0e2987d5-26a8-4816-bbf1-86e74daf5871_characterizeImage_945867_33.stderr parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4acb028ca933616cb75e2cf534002420 with result from task 830 parsl.dataflow.dflow DEBUG: Task 830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/354e48d5-9ec8-4bb4-ab94-62b4f5f62c6c_calibrate_945869_37.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/354e48d5-9ec8-4bb4-ab94-62b4f5f62c6c_calibrate_945869_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:32 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb89f9b649b93aa8b17cbc98a82e63d1 with result from task 996 parsl.dataflow.dflow DEBUG: Task 996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 997 has memoization hash 313389040ad634ca8f35e7fc282bfb3f parsl.dataflow.memoization INFO: Task 997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 991 parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/ffc29455-e1ae-4685-bc0e-dde333770849_characterizeImage_945757_60.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/ffc29455-e1ae-4685-bc0e-dde333770849_characterizeImage_945757_60.stderr parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a0c8b2e8-f660-4aa9-af2d-11a4ea1b41cb_isr_945757_60.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/a0c8b2e8-f660-4aa9-af2d-11a4ea1b41cb_isr_945757_60.stderr parsl.dataflow.dflow INFO: Task 993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 442665055de540c57a082457a7bc09c4 with result from task 993 parsl.dataflow.dflow DEBUG: Task 993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 994 has memoization hash 36204457327ec5e4efab65fd4064baeb parsl.dataflow.memoization INFO: Task 994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 992 parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/e9b885eb-ec59-4d8e-bfd4-57a2b8ea98b3_characterizeImage_945755_51.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/e9b885eb-ec59-4d8e-bfd4-57a2b8ea98b3_characterizeImage_945755_51.stderr parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e969c954-6cbd-4efc-befa-03726a0daf1f_isr_945755_51.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/e969c954-6cbd-4efc-befa-03726a0daf1f_isr_945755_51.stderr parsl.dataflow.dflow INFO: Task 984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a073ee7039b5f6b4fafe7e4acd20804e with result from task 984 parsl.dataflow.dflow DEBUG: Task 984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 985 has memoization hash c637ddc40217aa3c99cc0c7ba5a67253 parsl.dataflow.memoization INFO: Task 985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246228360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 993 parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/4718b404-6045-4215-aee8-bd38a8817346_characterizeImage_945757_28.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/4718b404-6045-4215-aee8-bd38a8817346_characterizeImage_945757_28.stderr parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/e41313bd-69be-4a06-a2ce-f477e7ccc6ad_isr_945757_28.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/e41313bd-69be-4a06-a2ce-f477e7ccc6ad_isr_945757_28.stderr parsl.dataflow.dflow INFO: Task 990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2df74aeb3c8caa28e252225eceaccfc with result from task 990 parsl.dataflow.dflow DEBUG: Task 990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 204a9f9ba73566d2e732869df3c48361 parsl.dataflow.memoization INFO: Task 991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 994 parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a7a495bc-10ba-4ef0-95ce-0273d6f2c9cc_characterizeImage_945757_25.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a7a495bc-10ba-4ef0-95ce-0273d6f2c9cc_characterizeImage_945757_25.stderr parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/557949c7-f5f5-49da-939f-f23e5c5c9039_isr_945757_25.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945757/557949c7-f5f5-49da-939f-f23e5c5c9039_isr_945757_25.stderr parsl.dataflow.dflow INFO: Task 978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 310b9ab9fac874add8214a8dd93fda91 with result from task 978 parsl.dataflow.dflow DEBUG: Task 978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 979 has memoization hash aa00da1d1a651c9a03a52ff6c0625635 parsl.dataflow.memoization INFO: Task 979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 995 parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/422061cf-a9c5-4de0-aa31-1d92caabccc6_characterizeImage_945756_58.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/422061cf-a9c5-4de0-aa31-1d92caabccc6_characterizeImage_945756_58.stderr parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/30b58ded-ceb5-43c0-bf36-c90df70140e8_isr_945756_58.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945756/30b58ded-ceb5-43c0-bf36-c90df70140e8_isr_945756_58.stderr parsl.dataflow.dflow INFO: Task 981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7a6bb29fcb3b315eb9800788c2761e9 with result from task 981 parsl.dataflow.dflow DEBUG: Task 981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 982 has memoization hash 80357df3c28238a48776038387ef2db7 parsl.dataflow.memoization INFO: Task 982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462280e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 996 parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/893ec485-1dd1-4ee0-984d-6e5596cb027f_characterizeImage_945869_38.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/893ec485-1dd1-4ee0-984d-6e5596cb027f_characterizeImage_945869_38.stderr parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e93ebc27-4720-4947-b56c-d87f83993ca4_isr_945869_38.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/e93ebc27-4720-4947-b56c-d87f83993ca4_isr_945869_38.stderr parsl.dataflow.dflow INFO: Task 975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b6e0a1183e8f950376ee9af29872911 with result from task 975 parsl.dataflow.dflow DEBUG: Task 975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 976 has memoization hash a2f71bfd17240aec0c94188eacac9f18 parsl.dataflow.memoization INFO: Task 976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 997 parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/7243d618-f8f2-42ed-92bf-4be4f72fea3b_characterizeImage_945869_17.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/7243d618-f8f2-42ed-92bf-4be4f72fea3b_characterizeImage_945869_17.stderr parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/c388f6a3-7af2-4569-bca7-e44694c52376_isr_945869_17.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945869/c388f6a3-7af2-4569-bca7-e44694c52376_isr_945869_17.stderr parsl.dataflow.dflow INFO: Task 987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ead752da727076941bc74c690efe2fd with result from task 987 parsl.dataflow.dflow DEBUG: Task 987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 988 has memoization hash 0db2d8fe4d3042002f13c81fd1b9f8d7 parsl.dataflow.memoization INFO: Task 988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 988 try 0 launched on executor multi with executor id 998 parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/804feb47-648a-44df-8e28-8155b44f1b25_characterizeImage_945755_48.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/804feb47-648a-44df-8e28-8155b44f1b25_characterizeImage_945755_48.stderr parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/0651664d-ca62-4342-a0df-6f7bc23b2d07_isr_945755_48.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/isr/945755/0651664d-ca62-4342-a0df-6f7bc23b2d07_isr_945755_48.stderr parsl.dataflow.dflow INFO: Task 1027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84479d27753fbbb84f6f92dbba4fe0c1 with result from task 1027 parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:33 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow DEBUG: Task 1027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash e5954beb65774f3a8844492020e9a771 parsl.dataflow.memoization INFO: Task 1028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246089120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 999 parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/e60a6e30-0f7f-4738-8625-ab4547aa0aee_calibrate_945755_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/e60a6e30-0f7f-4738-8625-ab4547aa0aee_calibrate_945755_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/3de3f01c-e506-4de1-8690-adf935e10122_characterizeImage_945755_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/3de3f01c-e506-4de1-8690-adf935e10122_characterizeImage_945755_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:33 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d89f7aba6f2d70d4690bb86d966bca6e with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b4913521-fdd4-4b6f-aacf-90c5fa2e6df6_calibrate_945869_55.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b4913521-fdd4-4b6f-aacf-90c5fa2e6df6_calibrate_945869_55.stderr parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7448c1a430eb7ba7cbe51bd5dac99cf4 with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash 57515d98dfe851d64f39dbaea132c94b parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 1000 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/34487c5c-03c5-4089-8dd7-c38b22e515c6_calibrate_945869_12.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/34487c5c-03c5-4089-8dd7-c38b22e515c6_calibrate_945869_12.stderr parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1aa08685-a257-47e2-9b48-f866df66125f_characterizeImage_945869_12.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1aa08685-a257-47e2-9b48-f866df66125f_characterizeImage_945869_12.stderr parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50d08e98d41dcce63097e316f9848499 with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/166489b5-7cb4-4b2c-b159-db335cbaf6c0_calibrate_945868_38.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/166489b5-7cb4-4b2c-b159-db335cbaf6c0_calibrate_945868_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:34 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b37f349446e3d84742653f449645a6b with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 094fd1c844f098aeb25ead10a42e889a parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246285080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 1001 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/bf5fee1e-4ad5-4bff-9f8a-52f7bc619d00_calibrate_945869_6.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/bf5fee1e-4ad5-4bff-9f8a-52f7bc619d00_calibrate_945869_6.stderr parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/add1eb9c-c90f-496e-9a55-95c0aa91a8f6_characterizeImage_945869_6.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/add1eb9c-c90f-496e-9a55-95c0aa91a8f6_characterizeImage_945869_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:34 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 211 parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c28db5f99bebd74bd52c6eb2c335d964 with result from task 233 parsl.dataflow.strategy DEBUG: Executor multi has 211 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:37:35 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/cd0eb8fb-ba62-42e8-9431-65f4100b9655_calibrate_945867_15.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/cd0eb8fb-ba62-42e8-9431-65f4100b9655_calibrate_945867_15.stderr parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd94ce7f8e2aaeaf881fd87145479354 with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/08bdd090-b345-4da0-92af-3b5280ca716c_calibrate_945869_57.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/08bdd090-b345-4da0-92af-3b5280ca716c_calibrate_945869_57.stderr 2024-03-13 18:37:35 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 1033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38fe8e63618a5530d5780c3696dc40fa with result from task 1033 parsl.dataflow.dflow DEBUG: Task 1033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1034 has memoization hash f677255cb9607d3610b68849b811ad84 parsl.dataflow.memoization INFO: Task 1034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246089760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1034 try 0 launched on executor multi with executor id 1002 parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/9063f4a8-1818-432c-beff-080f51aa6e8c_calibrate_945757_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/9063f4a8-1818-432c-beff-080f51aa6e8c_calibrate_945757_13.stderr parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/de5cbb61-fc15-4e55-a6b4-5d02bfae531f_characterizeImage_945757_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/de5cbb61-fc15-4e55-a6b4-5d02bfae531f_characterizeImage_945757_13.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f80f35937022ff053ec3fab1368f3d5 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2c853e0b-205e-485a-9230-c6bf696e332a_calibrate_945868_15.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2c853e0b-205e-485a-9230-c6bf696e332a_calibrate_945868_15.stderr parsl.dataflow.dflow INFO: Task 1015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5adc191007432261500d178234ca55c with result from task 1015 parsl.dataflow.dflow DEBUG: Task 1015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash 5be2e24c12c3ea95e006bade7ffb2334 parsl.dataflow.memoization INFO: Task 1016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460884a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 1003 parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/08b792bb-8305-4c61-b307-062d6e81a3fa_calibrate_945756_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/08b792bb-8305-4c61-b307-062d6e81a3fa_calibrate_945756_27.stderr parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/33d5e6ea-a671-4801-bb60-c83fe063bda0_characterizeImage_945756_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/33d5e6ea-a671-4801-bb60-c83fe063bda0_characterizeImage_945756_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:35 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c456671593a99ee7d286be7cff60161 with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash e0eca3d2a9ff29ab118b2d40ea49fd8e parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 1004 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/38ff1c9d-bf99-4b49-af52-395ce3904563_calibrate_945756_48.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/38ff1c9d-bf99-4b49-af52-395ce3904563_calibrate_945756_48.stderr parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/7196bb14-a6d5-41c8-9022-7cb1276bef4a_characterizeImage_945756_48.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/7196bb14-a6d5-41c8-9022-7cb1276bef4a_characterizeImage_945756_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:36 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eeef82c63f94b06d36c6e6fbac9e9cdb with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/0674f427-4cc2-4569-bb14-4cc4a7ac2432_calibrate_945867_48.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/0674f427-4cc2-4569-bb14-4cc4a7ac2432_calibrate_945867_48.stderr parsl.dataflow.dflow INFO: Task 1003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2d02c80f440deb94857eaac39a97e32 with result from task 1003 parsl.dataflow.dflow DEBUG: Task 1003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash a55de024501fe7f677a799efe53881c8 parsl.dataflow.memoization INFO: Task 1004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 1005 parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/e7700f5f-6c7a-447f-9246-fb4cb12324ea_calibrate_945756_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/e7700f5f-6c7a-447f-9246-fb4cb12324ea_calibrate_945756_20.stderr parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/02796bba-c3fd-4d86-9587-2a50638d7315_characterizeImage_945756_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/02796bba-c3fd-4d86-9587-2a50638d7315_characterizeImage_945756_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:36 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:37 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce01ef13938671b33bd29c1990df020f with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 51786b64b84ab15c7bbe90ae90f751b2 parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 1006 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/0a90b0b4-5371-4110-a8a5-fabaec267249_calibrate_945867_12.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/0a90b0b4-5371-4110-a8a5-fabaec267249_calibrate_945867_12.stderr parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/896517ee-ccbf-4c0f-80c2-058aea885946_characterizeImage_945867_12.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/896517ee-ccbf-4c0f-80c2-058aea885946_characterizeImage_945867_12.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 561e90360b35170a45296d32029d8a2d with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/5c326abd-23a8-4a24-ab94-726eef669a92_calibrate_945869_24.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/5c326abd-23a8-4a24-ab94-726eef669a92_calibrate_945869_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:37 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89d68c276cb2cde9c69996f37455d7ca with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/685381e8-6baa-418b-b28e-d86c9ef40c49_calibrate_945867_56.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/685381e8-6baa-418b-b28e-d86c9ef40c49_calibrate_945867_56.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e2bfcae06ca753de6dbf38acf27581e with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f32fd2f5-f3b7-42f7-bd39-e0d5c0984ece_calibrate_945755_24.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f32fd2f5-f3b7-42f7-bd39-e0d5c0984ece_calibrate_945755_24.stderr parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4edcd07475f0b0a3e09b6aea6446b4d2 with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8d20cc05-cc93-48e7-aa06-8ae431cad7f0_calibrate_945756_12.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8d20cc05-cc93-48e7-aa06-8ae431cad7f0_calibrate_945756_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:38 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5987e08fa443c826630346433877043c with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash 3a0747e4dec5ebd83cf421551cf3c4d6 parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 1007 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/46bdea0d-5577-445d-a0cb-67fffb2a508b_calibrate_945868_32.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/46bdea0d-5577-445d-a0cb-67fffb2a508b_calibrate_945868_32.stderr parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/1b3f9f7f-22a6-4180-bb3f-be829358a380_characterizeImage_945868_32.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/1b3f9f7f-22a6-4180-bb3f-be829358a380_characterizeImage_945868_32.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 677 try 0 failed parsl.dataflow.dflow INFO: Task 677 marked for retry parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b8098a9d-8d98-4b2f-9bb4-6c078a1f1de6_calibrate_945869_31.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b8098a9d-8d98-4b2f-9bb4-6c078a1f1de6_calibrate_945869_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash fcaa17cf1a301b25afed06dfb8c5e357 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462be200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 1 launched on executor multi with executor id 1008 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b8098a9d-8d98-4b2f-9bb4-6c078a1f1de6_calibrate_945869_31.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b8098a9d-8d98-4b2f-9bb4-6c078a1f1de6_calibrate_945869_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:38 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e636a591b47730b0482592c428d86ae5 with result from task 959 parsl.dataflow.dflow DEBUG: Task 959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/f576f2a4-8703-4368-b2b0-870d97e00ebf_calibrate_945756_23.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/f576f2a4-8703-4368-b2b0-870d97e00ebf_calibrate_945756_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:39 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 1045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e458afb1d4594504633d5c8e0378fa77 with result from task 1045 parsl.dataflow.dflow DEBUG: Task 1045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash 5040f0cadc5c6ec981b77a75aec53c57 parsl.dataflow.memoization INFO: Task 1046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 1009 parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/ce58f9eb-9df0-4716-98b8-379d6941a288_calibrate_945756_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/ce58f9eb-9df0-4716-98b8-379d6941a288_calibrate_945756_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/77384002-441d-4fdf-b778-4a370c0d4533_characterizeImage_945756_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/77384002-441d-4fdf-b778-4a370c0d4533_characterizeImage_945756_16.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0efb2bbb2405de08aa221e267948f560 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/89cf6cef-56fd-4fbe-bb9e-1bcb8d07f61b_calibrate_945757_15.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/89cf6cef-56fd-4fbe-bb9e-1bcb8d07f61b_calibrate_945757_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:39 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30ab1ac82e4eb6133c49589fbb9a7b70 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/afc2a44f-cb76-4dca-b642-185ea174ea84_calibrate_945868_49.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/afc2a44f-cb76-4dca-b642-185ea174ea84_calibrate_945868_49.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cb1de873bd90fc68d0a649871dbacf6 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f57be46e-99b0-4c7e-9691-60bc51265fcd_calibrate_945867_45.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f57be46e-99b0-4c7e-9691-60bc51265fcd_calibrate_945867_45.stderr parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fcf154a8c3fa6d3aa45e22a8f82c3c8 with result from task 791 parsl.dataflow.dflow DEBUG: Task 791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/0633eb18-9bfb-4124-9e71-71e8e5528ad1_calibrate_945756_1.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/0633eb18-9bfb-4124-9e71-71e8e5528ad1_calibrate_945756_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:40 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:40 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59bf962114080461c6014ffd0307737d with result from task 833 parsl.dataflow.dflow DEBUG: Task 833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d61263c8-155b-49e7-a52a-ee1f4ce8aa55_calibrate_945757_18.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d61263c8-155b-49e7-a52a-ee1f4ce8aa55_calibrate_945757_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:41 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2b024db7559228c0f48489820ae2486 with result from task 949 parsl.dataflow.dflow DEBUG: Task 949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 950 has memoization hash bb011fc2268f10f71ce31c2e848c29c5 parsl.dataflow.memoization INFO: Task 950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246229ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 950 try 0 launched on executor multi with executor id 1010 parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/56731c46-9d22-4424-b9c7-09658c87eb66_calibrate_945869_21.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/56731c46-9d22-4424-b9c7-09658c87eb66_calibrate_945869_21.stderr parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/cf509cec-6cae-40b4-a7ae-86c110c3df1e_characterizeImage_945869_21.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/cf509cec-6cae-40b4-a7ae-86c110c3df1e_characterizeImage_945869_21.stderr parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 610093f6f66164768b434cf546c2d9b6 with result from task 806 parsl.dataflow.dflow DEBUG: Task 806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/aa73124c-9116-48a2-8ae0-d2a7bcafe64e_calibrate_945755_50.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/aa73124c-9116-48a2-8ae0-d2a7bcafe64e_calibrate_945755_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:42 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62f9d9ec3f9e3494f9128443878b90fe with result from task 803 parsl.dataflow.dflow DEBUG: Task 803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/5d8fe282-7bab-4c32-85f1-6217ee2d4105_calibrate_945757_47.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/5d8fe282-7bab-4c32-85f1-6217ee2d4105_calibrate_945757_47.stderr parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78fc6194dea4e180526fce26cf8bdb3d with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash 95d37985d1d90c930ec9fbcaedb00a6c parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 1011 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/91f4fd29-e58b-47a4-bda6-f9ea99ae222b_calibrate_945755_38.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/91f4fd29-e58b-47a4-bda6-f9ea99ae222b_calibrate_945755_38.stderr parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/05b593f7-f5f2-4a14-bb29-93327950c08f_characterizeImage_945755_38.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/05b593f7-f5f2-4a14-bb29-93327950c08f_characterizeImage_945755_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:42 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 347d34457680375a95e340c27e683474 with result from task 902 parsl.dataflow.dflow DEBUG: Task 902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/9eb9759a-66e0-482a-8f28-4127e880b901_calibrate_945757_10.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/9eb9759a-66e0-482a-8f28-4127e880b901_calibrate_945757_10.stderr parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d786d0195d23e2063d063b900743bc5 with result from task 796 parsl.dataflow.dflow DEBUG: Task 796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash f26c5b48f859c8014b77a6444ab58f47 parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246345f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 1012 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/85de58a6-4deb-406e-8959-d16ff68f750d_calibrate_945869_7.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/85de58a6-4deb-406e-8959-d16ff68f750d_calibrate_945869_7.stderr parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/321beb0f-6910-4d56-b90a-a5753f96069d_characterizeImage_945869_7.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/321beb0f-6910-4d56-b90a-a5753f96069d_characterizeImage_945869_7.stderr parsl.dataflow.dflow INFO: Task 1000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c84f3a8d5aed1a5a2b1123fbc913d37 with result from task 1000 parsl.dataflow.dflow DEBUG: Task 1000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash baafa34944098a100de80652ead36bfc parsl.dataflow.memoization INFO: Task 1001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 1013 parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/adcb1911-15a1-41cf-8227-e93359d91ec4_calibrate_945755_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/adcb1911-15a1-41cf-8227-e93359d91ec4_calibrate_945755_33.stderr parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/41ba1c95-4313-4985-a4dd-9525d99cbfca_characterizeImage_945755_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/41ba1c95-4313-4985-a4dd-9525d99cbfca_characterizeImage_945755_33.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36c7d87840f04491b838bbc4c38c9811 with result from task 10 parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:43 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash f5ac7bb52eff722068655aa6c8e07fcb parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247916340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 1014 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/7bfc58b3-2563-4df9-ac7d-3492ec86ad4e_calibrate_945869_29.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/7bfc58b3-2563-4df9-ac7d-3492ec86ad4e_calibrate_945869_29.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bccd4b7b-0a93-4de6-b5aa-f4a96c21ab83_characterizeImage_945869_29.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bccd4b7b-0a93-4de6-b5aa-f4a96c21ab83_characterizeImage_945869_29.stderr parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b638c846a170a65cb348ff53b1f8fd4 with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/61d235fc-a98f-4161-9bd1-20bdc19321c1_calibrate_945755_49.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/61d235fc-a98f-4161-9bd1-20bdc19321c1_calibrate_945755_49.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 341 try 0 failed parsl.dataflow.dflow INFO: Task 341 marked for retry parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d4d6e0b3-e314-41e6-a7c3-84e79d043b2f_calibrate_945867_31.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d4d6e0b3-e314-41e6-a7c3-84e79d043b2f_calibrate_945867_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash d4c09f953c5e11695b53ba3ec73a6054 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 1 launched on executor multi with executor id 1015 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d4d6e0b3-e314-41e6-a7c3-84e79d043b2f_calibrate_945867_31.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d4d6e0b3-e314-41e6-a7c3-84e79d043b2f_calibrate_945867_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:43 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 910 try 0 failed parsl.dataflow.dflow INFO: Task 910 marked for retry parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/d142a2d6-48f3-4611-9dcf-338ff8834e5b_characterizeImage_945869_2.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/d142a2d6-48f3-4611-9dcf-338ff8834e5b_characterizeImage_945869_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash 3644f81ee40815484d53567637bfae60 parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 1 launched on executor multi with executor id 1016 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/d142a2d6-48f3-4611-9dcf-338ff8834e5b_characterizeImage_945869_2.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/d142a2d6-48f3-4611-9dcf-338ff8834e5b_characterizeImage_945869_2.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21a2af663eaca4d11e1eb5b1f3fef868 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/ed7d12c3-bc48-42e3-8775-49ff0e398239_calibrate_945867_17.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/ed7d12c3-bc48-42e3-8775-49ff0e398239_calibrate_945867_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:43 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50880a0290c04d1fea3bdf948e795b7f with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/878ab9fd-9483-4bd7-88ff-ee935f59573b_calibrate_945867_42.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/878ab9fd-9483-4bd7-88ff-ee935f59573b_calibrate_945867_42.stderr parsl.dataflow.dflow INFO: Task 1060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71cbfc696c74ba4087023dbc45b30494 with result from task 1060 parsl.dataflow.dflow DEBUG: Task 1060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1061 has memoization hash 5cc143e020b95fc4d64bda7d028ace33 parsl.dataflow.memoization INFO: Task 1061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1061 try 0 launched on executor multi with executor id 1017 parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/887fad17-a8d2-4f7b-b78f-466353bf3aca_calibrate_945867_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/887fad17-a8d2-4f7b-b78f-466353bf3aca_calibrate_945867_55.stderr parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4252745f-7c1a-4be4-8f57-84d027bf4d54_characterizeImage_945867_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4252745f-7c1a-4be4-8f57-84d027bf4d54_characterizeImage_945867_55.stderr parsl.dataflow.dflow INFO: Task 944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78d3076c9d2f008831be903ab9d76f23 with result from task 944 parsl.dataflow.dflow DEBUG: Task 944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/1888263a-48de-4176-ba61-a7a7bc025ab2_calibrate_945756_51.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/1888263a-48de-4176-ba61-a7a7bc025ab2_calibrate_945756_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:44 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:45 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 1024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 069fba163299b9017fde506024a2ff3e with result from task 1024 parsl.dataflow.dflow DEBUG: Task 1024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1025 has memoization hash 08b17006244c801236bdf9563f19aa35 parsl.dataflow.memoization INFO: Task 1025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246088e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1025 try 0 launched on executor multi with executor id 1018 parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/54daf6eb-f312-4e0f-a935-f362ab6fece2_calibrate_945756_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/54daf6eb-f312-4e0f-a935-f362ab6fece2_calibrate_945756_45.stderr parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/914fd3e6-6a10-4c14-a6d6-f392dc1e149e_characterizeImage_945756_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/914fd3e6-6a10-4c14-a6d6-f392dc1e149e_characterizeImage_945756_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:45 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 677e3dd4bc2b0787aa6bce8301d66b9f with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/76b00b96-b287-4462-9337-ecc13adee591_calibrate_945756_40.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/76b00b96-b287-4462-9337-ecc13adee591_calibrate_945756_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:46 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79968c11e7252038929392ce01f192c3 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1a02f2d3-77c1-4399-a941-aa05d4cca921_calibrate_945869_26.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1a02f2d3-77c1-4399-a941-aa05d4cca921_calibrate_945869_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:46 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6a54a3edc2c3139225dcefdd36342f4 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/4a125f34-3588-49eb-9a94-d33aa49b3ebb_calibrate_945869_4.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/4a125f34-3588-49eb-9a94-d33aa49b3ebb_calibrate_945869_4.stderr parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe9d0c4f9ba5241e512a3e750b24d75b with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 2e409fdb5335f0b8e22ae27c57cf62d5 parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246284a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 1019 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/33bffe38-41e4-410a-a8c9-68efa03ae3bd_calibrate_945869_45.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/33bffe38-41e4-410a-a8c9-68efa03ae3bd_calibrate_945869_45.stderr parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/8d6da76a-992f-4a7c-9cd8-c75ab94f85ff_characterizeImage_945869_45.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/8d6da76a-992f-4a7c-9cd8-c75ab94f85ff_characterizeImage_945869_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:47 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:47 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:48 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 796ad19ee600a826597a864184b90a5e with result from task 839 parsl.dataflow.dflow DEBUG: Task 839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/a96279ca-6e03-4bf7-9786-1cc25eb5bc7f_calibrate_945868_48.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/a96279ca-6e03-4bf7-9786-1cc25eb5bc7f_calibrate_945868_48.stderr parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d528b3dfb8820d88f2c6e197d170ea92 with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash e016d56a82bed4e02521a0d84ef6aa7e parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246403380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 1020 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/b6d9332c-3024-4d6c-9261-851748541032_calibrate_945757_36.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/b6d9332c-3024-4d6c-9261-851748541032_calibrate_945757_36.stderr parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/64be75ee-8814-4d46-bdee-87bfd4d830ad_characterizeImage_945757_36.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/64be75ee-8814-4d46-bdee-87bfd4d830ad_characterizeImage_945757_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:48 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8273c81d809a4198e84d11cf87420f10 with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash 2cf898c3a4d7dc898109a6b18ea064df parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 1021 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/2f8ea3ea-82df-4578-9632-2be297842c8a_calibrate_945757_21.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/2f8ea3ea-82df-4578-9632-2be297842c8a_calibrate_945757_21.stderr parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/2b27fb1e-ced2-43ae-93f7-547c73f4fcff_characterizeImage_945757_21.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/2b27fb1e-ced2-43ae-93f7-547c73f4fcff_characterizeImage_945757_21.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 910 try 1 failed parsl.dataflow.dflow ERROR: Task 910 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3644f81ee40815484d53567637bfae60 with result from task 910 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 911 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 911 try 0 failed parsl.dataflow.dflow INFO: Task 911 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/34e5af26-ca42-4b21-87d8-c49b10dc4b8d_calibrate_945869_2.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/34e5af26-ca42-4b21-87d8-c49b10dc4b8d_calibrate_945869_2.stderr parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/d142a2d6-48f3-4611-9dcf-338ff8834e5b_characterizeImage_945869_2.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/d142a2d6-48f3-4611-9dcf-338ff8834e5b_characterizeImage_945869_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:49 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:50 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f654e312884e3948d6fbde78fd995bc with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c6385baa-c413-42cf-be51-d88062b561be_calibrate_945756_14.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c6385baa-c413-42cf-be51-d88062b561be_calibrate_945756_14.stderr parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b709c528232012d6472ec7a4bf24037f with result from task 872 parsl.dataflow.dflow DEBUG: Task 872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/7685a7a1-c1d8-4cc5-be45-6d0f728ee91d_calibrate_945867_26.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/7685a7a1-c1d8-4cc5-be45-6d0f728ee91d_calibrate_945867_26.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 605 try 0 failed parsl.dataflow.dflow INFO: Task 605 marked for retry parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/22d1d061-a587-4cc3-8506-6319e7ebe162_calibrate_945869_11.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/22d1d061-a587-4cc3-8506-6319e7ebe162_calibrate_945869_11.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash c5f2e535a1a1a7141b8baa906d73fbe3 parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492462853a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 1 launched on executor multi with executor id 1022 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/22d1d061-a587-4cc3-8506-6319e7ebe162_calibrate_945869_11.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/22d1d061-a587-4cc3-8506-6319e7ebe162_calibrate_945869_11.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bc5458ad97a542649ec8f2619e108d4 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/985613c8-69a4-4c9e-9844-6661b5a1f0c3_calibrate_945868_19.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/985613c8-69a4-4c9e-9844-6661b5a1f0c3_calibrate_945868_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:50 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:51 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9346dc30e0cb5575844454c8b9015028 with result from task 784 parsl.dataflow.dflow DEBUG: Task 784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 785 has memoization hash c25fa1655e5af17d59719ef602d1f833 parsl.dataflow.memoization INFO: Task 785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 1023 parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/4bde8942-bc7b-4fa8-90c6-7d7c3cbc92da_calibrate_945869_28.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/4bde8942-bc7b-4fa8-90c6-7d7c3cbc92da_calibrate_945869_28.stderr parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/baff60b0-709f-4080-b56b-e02a6578393f_characterizeImage_945869_28.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/baff60b0-709f-4080-b56b-e02a6578393f_characterizeImage_945869_28.stderr parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d123e17e4a1d1b285a76fe4af7bffe08 with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79e8bb19-eaa6-4668-aeb3-7fd637994b80_calibrate_945868_36.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79e8bb19-eaa6-4668-aeb3-7fd637994b80_calibrate_945868_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:51 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f5e978a36c5a3821f83799bb58ebb04 with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0942ee44-be77-4467-8eea-cf0b171642f7_calibrate_945869_30.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0942ee44-be77-4467-8eea-cf0b171642f7_calibrate_945869_30.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a48ed67aaf639dca9a3ab88148afb560 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c6678285-664c-4f1d-89e2-7b793565b988_calibrate_945867_57.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c6678285-664c-4f1d-89e2-7b793565b988_calibrate_945867_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:52 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:52 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df89b8065fca401c95b985beb8e2ba19 with result from task 635 parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8a446210-c90e-4609-bc59-b822d7c808b0_calibrate_945868_5.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8a446210-c90e-4609-bc59-b822d7c808b0_calibrate_945868_5.stderr parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abc11387f4b851174bd4ce6c121ab5ae with result from task 896 parsl.dataflow.dflow DEBUG: Task 896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/e4599609-6468-419d-aaed-5f40c5ee4015_calibrate_945756_31.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/e4599609-6468-419d-aaed-5f40c5ee4015_calibrate_945756_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:53 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 976c97bce665253cae47b68690bebaae with result from task 887 parsl.dataflow.dflow DEBUG: Task 887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/0dbc9925-dc4c-48fe-9081-a34a253241f9_calibrate_945757_23.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/0dbc9925-dc4c-48fe-9081-a34a253241f9_calibrate_945757_23.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98d16abeee8c47c3f57a24c886ee8d25 with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/fa011f88-9499-4d18-a637-2cb6ca333262_calibrate_945755_41.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/fa011f88-9499-4d18-a637-2cb6ca333262_calibrate_945755_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:53 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:54 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:54 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:55 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 818 try 0 failed parsl.dataflow.dflow INFO: Task 818 marked for retry parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b9d1d390-934e-4c1e-9010-8a2e557e071e_calibrate_945868_31.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b9d1d390-934e-4c1e-9010-8a2e557e071e_calibrate_945868_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash aeb38d6db1fd7a16a96bb9771a9528a5 parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246347560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 1 launched on executor multi with executor id 1024 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b9d1d390-934e-4c1e-9010-8a2e557e071e_calibrate_945868_31.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b9d1d390-934e-4c1e-9010-8a2e557e071e_calibrate_945868_31.stderr parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 990a3ac9f8346597e515d5329aef6dd2 with result from task 809 parsl.dataflow.dflow DEBUG: Task 809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/6e396d61-2583-475c-b8b5-0902e98821a7_calibrate_945868_57.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/6e396d61-2583-475c-b8b5-0902e98821a7_calibrate_945868_57.stderr parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78a31f13174ab687c87a5d2eab529252 with result from task 781 parsl.dataflow.dflow DEBUG: Task 781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 782 has memoization hash c38473964773c13448dcb173e7298154 parsl.dataflow.memoization INFO: Task 782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246344540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 1025 parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/5bdb7516-c129-4555-a41e-998efdcab733_calibrate_945867_44.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/5bdb7516-c129-4555-a41e-998efdcab733_calibrate_945867_44.stderr parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2fef4c39-3e7a-4a22-a8eb-62992a8c3a73_characterizeImage_945867_44.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/2fef4c39-3e7a-4a22-a8eb-62992a8c3a73_characterizeImage_945867_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:55 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5 try 0 failed parsl.dataflow.dflow INFO: Task 5 marked for retry parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/95678e94-3e5f-4805-804a-55a1881d9410_calibrate_945868_11.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/95678e94-3e5f-4805-804a-55a1881d9410_calibrate_945868_11.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 0d76ac2f13d8970266d0abc4a3dc6c07 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149247915da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 1 launched on executor multi with executor id 1026 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/95678e94-3e5f-4805-804a-55a1881d9410_calibrate_945868_11.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/95678e94-3e5f-4805-804a-55a1881d9410_calibrate_945868_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:56 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72578fc0965c9f804eafffb4309ffbd5 with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash d61b43c4fece7ad1a65eb75bed7fe5a5 parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 1027 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f01de854-a0ad-4055-b58a-388488cf4f5d_calibrate_945868_10.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f01de854-a0ad-4055-b58a-388488cf4f5d_calibrate_945868_10.stderr parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/50f4c3ce-5ed6-4fc7-809b-3e0a9a3e2cf2_characterizeImage_945868_10.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/50f4c3ce-5ed6-4fc7-809b-3e0a9a3e2cf2_characterizeImage_945868_10.stderr parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf6cf1fc1016e2043253c8da5f69e9ab with result from task 889 parsl.dataflow.dflow DEBUG: Task 889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 890 has memoization hash 895b872d89f0d3e887f9d036ab910f7a parsl.dataflow.memoization INFO: Task 890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 1028 parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f40b16bb-4c14-4f49-9e12-3c100790dbf5_calibrate_945868_45.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f40b16bb-4c14-4f49-9e12-3c100790dbf5_calibrate_945868_45.stderr parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/6863ad2e-2c66-45c3-a7fa-61a942f5a56c_characterizeImage_945868_45.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/6863ad2e-2c66-45c3-a7fa-61a942f5a56c_characterizeImage_945868_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:56 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 422c6bce204de6a3bbd00a86368ea6b8 with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash 83ac01ecd08085d03f3764fb26512ab9 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 1029 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c8b44883-2f53-444e-89fd-1178198620c9_calibrate_945756_34.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c8b44883-2f53-444e-89fd-1178198620c9_calibrate_945756_34.stderr parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/e0d88d16-8b49-4665-886f-1ddd1cd122bb_characterizeImage_945756_34.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/e0d88d16-8b49-4665-886f-1ddd1cd122bb_characterizeImage_945756_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:57 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42a92b71775f001ef49d0da0cd60abb5 with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash c11632f5499dc1c691bad509e5cac643 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246403060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 1030 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8e584c07-7629-4ab5-8710-de14d4431b9e_calibrate_945756_43.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8e584c07-7629-4ab5-8710-de14d4431b9e_calibrate_945756_43.stderr parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/0e11714c-a0ca-4da3-aaf5-42594f854e7e_characterizeImage_945756_43.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/0e11714c-a0ca-4da3-aaf5-42594f854e7e_characterizeImage_945756_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:57 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e133d6fa01181938c29b2c1a4f2ace1 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/e8cf0312-c471-4c5f-9c62-fae4e7b2eac2_calibrate_945869_62.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/e8cf0312-c471-4c5f-9c62-fae4e7b2eac2_calibrate_945869_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:58 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63677406e0c8968e68c7c980ac932158 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/cd88c30e-fa51-4375-8432-3591efba05e7_calibrate_945869_13.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/cd88c30e-fa51-4375-8432-3591efba05e7_calibrate_945869_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:59 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0db2d8fe4d3042002f13c81fd1b9f8d7 with result from task 988 parsl.dataflow.dflow DEBUG: Task 988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 989 has memoization hash 39a31b3237e1a0ff0bcade57ec07184c parsl.dataflow.memoization INFO: Task 989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 1031 parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7e3c9590-bf95-4316-adcc-bb21a9f9ab52_calibrate_945755_48.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7e3c9590-bf95-4316-adcc-bb21a9f9ab52_calibrate_945755_48.stderr parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/804feb47-648a-44df-8e28-8155b44f1b25_characterizeImage_945755_48.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/804feb47-648a-44df-8e28-8155b44f1b25_characterizeImage_945755_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:37:59 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60a8e5ae4d83bd435cc76c17b85512c4 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/29fb3726-13b4-41b4-b86b-b3491f0de1bf_calibrate_945868_43.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/29fb3726-13b4-41b4-b86b-b3491f0de1bf_calibrate_945868_43.stderr parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7132407521d9775c491cdaebcdb0d5a7 with result from task 757 parsl.dataflow.dflow DEBUG: Task 757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 758 has memoization hash e73f5a8bbff7714c19f66689e6dd0351 parsl.dataflow.memoization INFO: Task 758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246312f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 1032 parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/3b7eb95b-8476-4e6b-9de5-e26593f4958c_calibrate_945867_47.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/3b7eb95b-8476-4e6b-9de5-e26593f4958c_calibrate_945867_47.stderr parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/a002c301-c0b3-4b5f-a43a-4875774ad2d7_characterizeImage_945867_47.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/a002c301-c0b3-4b5f-a43a-4875774ad2d7_characterizeImage_945867_47.stderr parsl.dataflow.dflow INFO: Task 1030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b08bd2e6107341c5f6499d99d72ae73 with result from task 1030 parsl.dataflow.dflow DEBUG: Task 1030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash 8818784a5b34a7cc0b7326bb57f2a060 parsl.dataflow.memoization INFO: Task 1031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246089440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 1033 parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/3da6387d-05a0-47c6-8d07-a7eecfac4a7c_calibrate_945757_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/3da6387d-05a0-47c6-8d07-a7eecfac4a7c_calibrate_945757_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dd804a8e-6844-41c7-b599-eb37eee39a0c_characterizeImage_945757_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/dd804a8e-6844-41c7-b599-eb37eee39a0c_characterizeImage_945757_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:00 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7deb2c315eab168f540595927b621fb6 with result from task 689 parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/e61c675d-82c2-4598-b823-618d1d6e5bcc_calibrate_945869_32.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/e61c675d-82c2-4598-b823-618d1d6e5bcc_calibrate_945869_32.stderr parsl.dataflow.dflow INFO: Task 1028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5954beb65774f3a8844492020e9a771 with result from task 1028 parsl.dataflow.dflow DEBUG: Task 1028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/e60a6e30-0f7f-4738-8625-ab4547aa0aee_calibrate_945755_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/e60a6e30-0f7f-4738-8625-ab4547aa0aee_calibrate_945755_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:00 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37849c912ccad83384d1609034a29f61 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c88f99dd-765b-485a-97ed-b7a7a62f6e4c_calibrate_945867_21.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c88f99dd-765b-485a-97ed-b7a7a62f6e4c_calibrate_945867_21.stderr parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f0b7d4b611083665b48b0ae12972596 with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash 84a8ee38703cf23ce76c0161af0b315b parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 1034 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/8e106d20-2da4-4238-8eed-f3f7d42430fa_calibrate_945867_34.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/8e106d20-2da4-4238-8eed-f3f7d42430fa_calibrate_945867_34.stderr parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/99ef614b-8bfb-44f6-9d52-ea0ede28e218_characterizeImage_945867_34.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/99ef614b-8bfb-44f6-9d52-ea0ede28e218_characterizeImage_945867_34.stderr parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 528caa1c4f6db5164d9a90433d67906f with result from task 748 parsl.dataflow.dflow DEBUG: Task 748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash fd43db89902db3ea11d8591ab9f47f1d parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246312480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 1035 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c3456b46-95a1-4e62-b746-5763bcb5e779_calibrate_945867_53.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c3456b46-95a1-4e62-b746-5763bcb5e779_calibrate_945867_53.stderr parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e67a3eaf-de9e-4df2-9d18-64888ad2263c_characterizeImage_945867_53.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e67a3eaf-de9e-4df2-9d18-64888ad2263c_characterizeImage_945867_53.stderr parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c999a2caa8510f5286c865b12d87dd3b with result from task 919 parsl.dataflow.dflow DEBUG: Task 919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 920 has memoization hash baa4a222b2746fec25a532ba10592491 parsl.dataflow.memoization INFO: Task 920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 1036 parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/74355852-3cd3-46ed-a6b3-77d51dbded55_calibrate_945755_44.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/74355852-3cd3-46ed-a6b3-77d51dbded55_calibrate_945755_44.stderr parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/4cb5e501-c590-4a87-aaa2-95eb8b16c3b7_characterizeImage_945755_44.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/4cb5e501-c590-4a87-aaa2-95eb8b16c3b7_characterizeImage_945755_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:01 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f1a576176e7cb51aebaedb6bb74003a with result from task 934 parsl.dataflow.dflow DEBUG: Task 934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 935 has memoization hash 2dd40bf7551e4ddfba12d09353acf113 parsl.dataflow.memoization INFO: Task 935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246228e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 1037 parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/a4761586-38ff-48d7-9e23-eca5d03ac51d_calibrate_945756_6.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/a4761586-38ff-48d7-9e23-eca5d03ac51d_calibrate_945756_6.stderr parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/3e11dade-18b6-4033-9c02-ce2a97630e34_characterizeImage_945756_6.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/3e11dade-18b6-4033-9c02-ce2a97630e34_characterizeImage_945756_6.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c95df3e760f27ee03f7251297e259b30 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/6070f19a-b36f-4f83-b1b6-62179096ff15_calibrate_945867_9.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/6070f19a-b36f-4f83-b1b6-62179096ff15_calibrate_945867_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:01 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:01 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96215f0dc793b03c75ba98c360485b1d with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79aa65e0-6a2c-4c92-82c4-e1477a52a589_calibrate_945868_3.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79aa65e0-6a2c-4c92-82c4-e1477a52a589_calibrate_945868_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:02 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0032340e69cdd44ae76fe9194141f33e with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/ca39a6b5-520b-4d44-9ee2-82af7190bc26_calibrate_945869_20.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/ca39a6b5-520b-4d44-9ee2-82af7190bc26_calibrate_945869_20.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 647ba9b12924510a65d93acd31ed651a with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/f79b6636-81df-4ba7-ae84-19082a3ce231_calibrate_945756_29.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/f79b6636-81df-4ba7-ae84-19082a3ce231_calibrate_945756_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:02 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd247231f655fa64222d64cc27ce9a0e with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash ef3ab0473380b10cb244dc9718abc399 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492464036a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 1038 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/4f99a9d2-ce67-45a8-a913-70a90c12158a_calibrate_945756_44.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/4f99a9d2-ce67-45a8-a913-70a90c12158a_calibrate_945756_44.stderr parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4607e1c1-2060-4625-9911-452b33744ea8_characterizeImage_945756_44.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4607e1c1-2060-4625-9911-452b33744ea8_characterizeImage_945756_44.stderr parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd520330ed4e3f7a3505bd1a52fcb386 with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash bc6cea0faccc541077b923a678558166 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 1039 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/9fffcef9-59fa-4217-a21c-1951e8221037_calibrate_945867_59.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/9fffcef9-59fa-4217-a21c-1951e8221037_calibrate_945867_59.stderr parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/8490f1af-9c55-4453-90dd-2f5fd8983244_characterizeImage_945867_59.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/8490f1af-9c55-4453-90dd-2f5fd8983244_characterizeImage_945867_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:03 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb87fd6c4c958e06ad7169dcc7a962ea with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/166854bf-b992-4661-bc84-d27716f85bd7_calibrate_945868_12.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/166854bf-b992-4661-bc84-d27716f85bd7_calibrate_945868_12.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 212b4c51b323b73fdaf9cc98deb1cd59 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2aa80c99-ae0a-4aed-b685-3bbdb9ce45f9_calibrate_945868_23.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2aa80c99-ae0a-4aed-b685-3bbdb9ce45f9_calibrate_945868_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:03 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2f3c8ed0febd09603613fd7f8dfdf00 with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/dc32c03b-2834-4644-9c2b-b73b8e1569c7_calibrate_945869_3.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/dc32c03b-2834-4644-9c2b-b73b8e1569c7_calibrate_945869_3.stderr parsl.dataflow.dflow INFO: Task 1115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aaa7e872420b926aa2fcc3aa03b9f3d5 with result from task 1115 parsl.dataflow.dflow DEBUG: Task 1115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f614a9ae-e880-4b24-92d0-0198983d2d08_calibrate_945867_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f614a9ae-e880-4b24-92d0-0198983d2d08_calibrate_945867_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:04 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:04 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a70f9ac26a109b60c561e8aa77b8d2fd with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash b17f571d8129f6a7c1bbaf1ac8af3e9f parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 1040 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/7be7d14d-3921-4368-af4b-a5e7a4cd7f6a_calibrate_945868_26.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/7be7d14d-3921-4368-af4b-a5e7a4cd7f6a_calibrate_945868_26.stderr parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/f44918c0-0de1-430e-aa7d-dad94a11ef8c_characterizeImage_945868_26.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/f44918c0-0de1-430e-aa7d-dad94a11ef8c_characterizeImage_945868_26.stderr parsl.dataflow.dflow INFO: Task 1034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f677255cb9607d3610b68849b811ad84 with result from task 1034 parsl.dataflow.dflow DEBUG: Task 1034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/9063f4a8-1818-432c-beff-080f51aa6e8c_calibrate_945757_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/9063f4a8-1818-432c-beff-080f51aa6e8c_calibrate_945757_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:04 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:04 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbc4f9f61657d04e5e504f8a2f20a1ec with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash 4ca82012e96ec8f372ac91f2e771f4e0 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 1041 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/ec6d8339-706b-46c4-8bb5-5bf3e47be1ea_calibrate_945757_37.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/ec6d8339-706b-46c4-8bb5-5bf3e47be1ea_calibrate_945757_37.stderr parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/392f5067-62b1-472f-8c0e-dd39bcec876e_characterizeImage_945757_37.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/392f5067-62b1-472f-8c0e-dd39bcec876e_characterizeImage_945757_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:04 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0eca3d2a9ff29ab118b2d40ea49fd8e with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/38ff1c9d-bf99-4b49-af52-395ce3904563_calibrate_945756_48.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/38ff1c9d-bf99-4b49-af52-395ce3904563_calibrate_945756_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:05 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:05 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95d37985d1d90c930ec9fbcaedb00a6c with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/91f4fd29-e58b-47a4-bda6-f9ea99ae222b_calibrate_945755_38.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/91f4fd29-e58b-47a4-bda6-f9ea99ae222b_calibrate_945755_38.stderr parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60d32aafe849ae04e8e04f031b0d48da with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d5318749-75a6-4c1e-95ad-02dc1ea33a5a_calibrate_945867_39.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d5318749-75a6-4c1e-95ad-02dc1ea33a5a_calibrate_945867_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:05 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc07d7138e880fed30bf241a523a21cc with result from task 866 parsl.dataflow.dflow DEBUG: Task 866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/e36c9c77-2235-4e69-8bef-75bdd9ca8168_calibrate_945868_54.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/e36c9c77-2235-4e69-8bef-75bdd9ca8168_calibrate_945868_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:05 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef3656a9b216e74afcdb67120469463c with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8e306d81-f97f-43ce-bb9b-12cf9bc33f15_calibrate_945868_35.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8e306d81-f97f-43ce-bb9b-12cf9bc33f15_calibrate_945868_35.stderr parsl.dataflow.dflow INFO: Task 1016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5be2e24c12c3ea95e006bade7ffb2334 with result from task 1016 parsl.dataflow.dflow DEBUG: Task 1016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/08b792bb-8305-4c61-b307-062d6e81a3fa_calibrate_945756_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/08b792bb-8305-4c61-b307-062d6e81a3fa_calibrate_945756_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:06 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f2deb96604ba65551005ab6cb8c0477 with result from task 1006 parsl.dataflow.dflow DEBUG: Task 1006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash 0f031925461ed928d33434f9a0eed2f8 parsl.dataflow.memoization INFO: Task 1007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 1042 parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/927284b5-14f4-476b-80e9-e205145922aa_calibrate_945869_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/927284b5-14f4-476b-80e9-e205145922aa_calibrate_945869_59.stderr parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/f9cd966e-24a3-493c-a033-5b1d35811293_characterizeImage_945869_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/f9cd966e-24a3-493c-a033-5b1d35811293_characterizeImage_945869_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:06 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:06 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd8ea1a6f4b2ed65bced04c83ec6d4a4 with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/38174a41-78b0-4844-b9ae-68504bfadf37_calibrate_945868_34.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/38174a41-78b0-4844-b9ae-68504bfadf37_calibrate_945868_34.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 677 try 1 failed parsl.dataflow.dflow ERROR: Task 677 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry fcaa17cf1a301b25afed06dfb8c5e357 with result from task 677 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b8098a9d-8d98-4b2f-9bb4-6c078a1f1de6_calibrate_945869_31.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/b8098a9d-8d98-4b2f-9bb4-6c078a1f1de6_calibrate_945869_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:06 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:07 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:07 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 993f308a478d4ddee21bc50462fa98e7 with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 9fb5d46cf2b78df01bd2bf282c5d838e parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 1043 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/979bf86a-4002-4d08-bd69-14a4b46c5f89_calibrate_945756_24.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/979bf86a-4002-4d08-bd69-14a4b46c5f89_calibrate_945756_24.stderr parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/fd14c9c4-9c6b-4b39-bf76-6f11d49dec5f_characterizeImage_945756_24.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/fd14c9c4-9c6b-4b39-bf76-6f11d49dec5f_characterizeImage_945756_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:07 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:07 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:08 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c15e24ec3dae566aed89b95b9ca7a755 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/206e2b22-6331-4304-824e-f3fe1bc0bb2a_calibrate_945869_8.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/206e2b22-6331-4304-824e-f3fe1bc0bb2a_calibrate_945869_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:08 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:08 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:08 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:08 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:09 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ee65e2852b2bea720da4c89487e3a2c with result from task 773 parsl.dataflow.dflow DEBUG: Task 773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/67a21569-a691-4848-a243-54711cff21a7_calibrate_945868_22.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/67a21569-a691-4848-a243-54711cff21a7_calibrate_945868_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:09 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 005f6d5f0e9227c8aad046fc51d79bca with result from task 787 parsl.dataflow.dflow DEBUG: Task 787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 788 has memoization hash 9e1837ac7b963c2ef2d93b608d24392c parsl.dataflow.memoization INFO: Task 788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246345620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 1044 parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/d6f99150-b0e9-4c14-8301-00558827dcc9_calibrate_945869_44.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/d6f99150-b0e9-4c14-8301-00558827dcc9_calibrate_945869_44.stderr parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/b3bb46ea-d941-4c5e-9770-dde2a8f71612_characterizeImage_945869_44.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/b3bb46ea-d941-4c5e-9770-dde2a8f71612_characterizeImage_945869_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:09 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:09 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c95612bb0957c98d6aed3eef1efd656 with result from task 1082 parsl.dataflow.dflow DEBUG: Task 1082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/5dbdb2c9-25fc-48c8-889d-da65352d0a79_calibrate_945868_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/5dbdb2c9-25fc-48c8-889d-da65352d0a79_calibrate_945868_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:10 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:10 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12ddc4fb71cd7d1cae3ac97fbe30f680 with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash e89c8bc9c33098ebcaca8bc54564b3d4 parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 1045 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/9ee28858-35bc-4268-8505-3581e5f87efb_calibrate_945868_47.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/9ee28858-35bc-4268-8505-3581e5f87efb_calibrate_945868_47.stderr parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/6a75f1ae-9a2e-4cff-a581-9c1d5da7daa1_characterizeImage_945868_47.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/6a75f1ae-9a2e-4cff-a581-9c1d5da7daa1_characterizeImage_945868_47.stderr parsl.dataflow.dflow INFO: Task 1048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1ba77786057c5b5350fd5df5491bcc6 with result from task 1048 parsl.dataflow.dflow DEBUG: Task 1048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash 15fa4ba761f3315d637e55f6eb47982c parsl.dataflow.memoization INFO: Task 1049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 1046 parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/3bdc4835-6352-48a2-b9aa-7cb492c3784e_calibrate_945756_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/3bdc4835-6352-48a2-b9aa-7cb492c3784e_calibrate_945756_4.stderr parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4121fd82-9e81-4930-b739-e9c03ec4c8d8_characterizeImage_945756_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/4121fd82-9e81-4930-b739-e9c03ec4c8d8_characterizeImage_945756_4.stderr parsl.dataflow.dflow INFO: Task 1069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2496cd3e726eba274090c149d582d4e with result from task 1069 parsl.dataflow.dflow DEBUG: Task 1069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1070 has memoization hash 414004bbacf4db0d425799df062bb553 parsl.dataflow.memoization INFO: Task 1070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1070 try 0 launched on executor multi with executor id 1047 parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ec6d2474-bd91-47ee-80ea-140d0e6daea8_calibrate_945868_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ec6d2474-bd91-47ee-80ea-140d0e6daea8_calibrate_945868_59.stderr parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/043316c7-0d58-412d-b3f9-aa511f7c47aa_characterizeImage_945868_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/043316c7-0d58-412d-b3f9-aa511f7c47aa_characterizeImage_945868_59.stderr parsl.dataflow.dflow INFO: Task 1072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b9a3a049d7e24497a2afb529882ac28 with result from task 1072 parsl.dataflow.dflow DEBUG: Task 1072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1073 has memoization hash 017c06f8ab1761f4892b1c81e89d966a parsl.dataflow.memoization INFO: Task 1073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1073 try 0 launched on executor multi with executor id 1048 parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/18cfcdc5-8abb-4366-b837-259bf9cbd6ef_calibrate_945867_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/18cfcdc5-8abb-4366-b837-259bf9cbd6ef_calibrate_945867_4.stderr parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/0ff7d598-bbcc-4555-8044-8112c607c7f5_characterizeImage_945867_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/0ff7d598-bbcc-4555-8044-8112c607c7f5_characterizeImage_945867_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:10 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:10 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:11 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 341 try 1 failed parsl.dataflow.dflow ERROR: Task 341 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d4c09f953c5e11695b53ba3ec73a6054 with result from task 341 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d4d6e0b3-e314-41e6-a7c3-84e79d043b2f_calibrate_945867_31.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d4d6e0b3-e314-41e6-a7c3-84e79d043b2f_calibrate_945867_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:11 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ea559879396985f63a4664612659b00 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 945696b37367f484ae3a875653966480 parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492464009a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 1049 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/f1a31a94-24f6-43ad-ada1-ac08f1b54d9f_calibrate_945869_52.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/f1a31a94-24f6-43ad-ada1-ac08f1b54d9f_calibrate_945869_52.stderr parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/305d1d01-e448-4b9c-aab5-d7490be37fe1_characterizeImage_945869_52.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/305d1d01-e448-4b9c-aab5-d7490be37fe1_characterizeImage_945869_52.stderr parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c3bc5109c6484adca0efb8f44910974 with result from task 656 parsl.dataflow.dflow DEBUG: Task 656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/693989fd-a547-416f-9677-27efeef008e8_calibrate_945757_19.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/693989fd-a547-416f-9677-27efeef008e8_calibrate_945757_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:11 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:11 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08b17006244c801236bdf9563f19aa35 with result from task 1025 parsl.dataflow.dflow DEBUG: Task 1025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/54daf6eb-f312-4e0f-a935-f362ab6fece2_calibrate_945756_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/54daf6eb-f312-4e0f-a935-f362ab6fece2_calibrate_945756_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:12 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:12 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 145 parsl.dataflow.strategy DEBUG: Executor multi has 145 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:38:12 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:38:12 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:12 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 1078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f919f07e159d973a2a477fe7f444daf7 with result from task 1078 parsl.dataflow.dflow DEBUG: Task 1078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash fc45a80aff1728bca76455cca2a09be6 parsl.dataflow.memoization INFO: Task 1079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1079 try 0 launched on executor multi with executor id 1050 parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c99ee9b1-e7d9-4f83-96e3-f08da9e3bc43_calibrate_945867_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c99ee9b1-e7d9-4f83-96e3-f08da9e3bc43_calibrate_945867_51.stderr parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/d2038f1a-053c-4fdd-a3bf-ae24f4d667ca_characterizeImage_945867_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/d2038f1a-053c-4fdd-a3bf-ae24f4d667ca_characterizeImage_945867_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:13 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:13 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:13 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:13 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 1004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a55de024501fe7f677a799efe53881c8 with result from task 1004 parsl.dataflow.dflow DEBUG: Task 1004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/e7700f5f-6c7a-447f-9246-fb4cb12324ea_calibrate_945756_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/e7700f5f-6c7a-447f-9246-fb4cb12324ea_calibrate_945756_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:13 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a0c8ef3a2c16834d5ce3997f5ea0d03 with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 765d37a9e712c334493c8bbf6ee4b5fe parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 1051 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/0277ca91-96b2-4093-a122-8ba2708b68bd_calibrate_945867_18.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/0277ca91-96b2-4093-a122-8ba2708b68bd_calibrate_945867_18.stderr parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/493d2468-e857-4177-9299-99980dc497bf_characterizeImage_945867_18.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/493d2468-e857-4177-9299-99980dc497bf_characterizeImage_945867_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:14 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:14 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:14 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:14 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e50437b61e8c7cc20fa51e044f6252b with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 2b467daf806e38fbdb475b83f1ca5732 parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492464025c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 1052 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/25091d7d-6ade-48d5-ac04-c82b191d82f4_calibrate_945869_22.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/25091d7d-6ade-48d5-ac04-c82b191d82f4_calibrate_945869_22.stderr parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/7c2c1d31-49c1-428f-b8ea-e6ecc4877418_characterizeImage_945869_22.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/7c2c1d31-49c1-428f-b8ea-e6ecc4877418_characterizeImage_945869_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:15 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:15 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:15 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79fa7bd4e7abac569d961309beb00dc3 with result from task 836 parsl.dataflow.dflow DEBUG: Task 836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/bce82a5f-c68f-4361-a9c7-9d30aa13b4fd_calibrate_945867_62.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/bce82a5f-c68f-4361-a9c7-9d30aa13b4fd_calibrate_945867_62.stderr parsl.dataflow.dflow INFO: Task 1021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9420499e591ba997fbcf8786a6b54635 with result from task 1021 parsl.dataflow.dflow DEBUG: Task 1021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash 586f571b5c65247b8d329cd5a814d82c parsl.dataflow.memoization INFO: Task 1022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246088ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 1053 parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/e791ca8f-5a1a-41ba-bb10-4c2101541e0d_calibrate_945868_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/e791ca8f-5a1a-41ba-bb10-4c2101541e0d_calibrate_945868_39.stderr parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/dd53d6b7-31a9-4509-a0e4-d28e512dee5c_characterizeImage_945868_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/dd53d6b7-31a9-4509-a0e4-d28e512dee5c_characterizeImage_945868_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:15 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:16 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:16 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13ce0e34e9c50adb90a0d5636392ca87 with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/1a545bee-a5fe-4382-9d79-b63522e70f2c_calibrate_945868_8.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/1a545bee-a5fe-4382-9d79-b63522e70f2c_calibrate_945868_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:16 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:16 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e4bf46307f7d7a49a8b8a77f9c2b9cd with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/b5eb9ee3-575f-4c62-8a00-3793c5f7580b_calibrate_945757_42.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/b5eb9ee3-575f-4c62-8a00-3793c5f7580b_calibrate_945757_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:17 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:17 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:17 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c40734718c2aedaf3d081a33e0be42f1 with result from task 766 parsl.dataflow.dflow DEBUG: Task 766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash 591b371a4425ea2077ad027b5b2827b9 parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246313880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 1054 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f80edba8-0bc0-44d5-9957-d11f654800d6_calibrate_945755_25.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f80edba8-0bc0-44d5-9957-d11f654800d6_calibrate_945755_25.stderr parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/c400edc2-1a4d-49c6-8556-610471e659c6_characterizeImage_945755_25.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/c400edc2-1a4d-49c6-8556-610471e659c6_characterizeImage_945755_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:17 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:17 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c428652d6267f44f3cf1952d33dd5f04 with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/63b6da71-72af-47be-9e5f-5df6c0492fef_calibrate_945867_60.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/63b6da71-72af-47be-9e5f-5df6c0492fef_calibrate_945867_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:18 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:18 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:18 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:18 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 065bfe60221d6665a68336b42b2d788a with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d2923cbf-b1a9-4dc1-8068-f14d10975357_calibrate_945867_49.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d2923cbf-b1a9-4dc1-8068-f14d10975357_calibrate_945867_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:19 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:19 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:19 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 420df8fc323f804741ad24a9ab6b720f with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash d61ddcffdaf1483d8fd5ebd2e3d1f9ac parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246393ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 1055 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/cafd6624-5ae5-417a-b9d1-42f35222d81d_calibrate_945868_7.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/cafd6624-5ae5-417a-b9d1-42f35222d81d_calibrate_945868_7.stderr parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/2db9bd7d-28b7-4555-a566-04673aa088b5_characterizeImage_945868_7.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/2db9bd7d-28b7-4555-a566-04673aa088b5_characterizeImage_945868_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:19 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21d77a6a61bf4e319d475a8cc23c5269 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/391b3755-316b-41fc-bd39-582cc3f06cda_calibrate_945867_38.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/391b3755-316b-41fc-bd39-582cc3f06cda_calibrate_945867_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:19 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 226339ac707ffd722b72063a9afadad8 with result from task 827 parsl.dataflow.dflow DEBUG: Task 827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2464f8e4-f2ec-4669-962e-d03f694b9849_calibrate_945868_46.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2464f8e4-f2ec-4669-962e-d03f694b9849_calibrate_945868_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:20 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 1001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baafa34944098a100de80652ead36bfc with result from task 1001 parsl.dataflow.dflow DEBUG: Task 1001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/adcb1911-15a1-41cf-8227-e93359d91ec4_calibrate_945755_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/adcb1911-15a1-41cf-8227-e93359d91ec4_calibrate_945755_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:20 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e016d56a82bed4e02521a0d84ef6aa7e with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/b6d9332c-3024-4d6c-9261-851748541032_calibrate_945757_36.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/b6d9332c-3024-4d6c-9261-851748541032_calibrate_945757_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:20 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:20 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:21 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:21 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:21 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73c2f09baf2e9a8d3d1b10a950ac0b84 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f13a45f9-7e40-43d6-8155-e1abcb04b82c_calibrate_945868_56.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f13a45f9-7e40-43d6-8155-e1abcb04b82c_calibrate_945868_56.stderr parsl.dataflow.dflow INFO: Task 1009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 971aa82212cfbe5f267bae096c11b9d0 with result from task 1009 parsl.dataflow.dflow DEBUG: Task 1009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash 6558664b2fbd1473ae1525621fe4433a parsl.dataflow.memoization INFO: Task 1010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 1056 parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/fd2b3ecb-825c-418e-be28-fd9d58953737_calibrate_945756_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/fd2b3ecb-825c-418e-be28-fd9d58953737_calibrate_945756_25.stderr parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f06fa351-9a99-482e-a544-1c019d6afbe7_characterizeImage_945756_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/f06fa351-9a99-482e-a544-1c019d6afbe7_characterizeImage_945756_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:21 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d19d50962bf2b1f77aaf198a6c3eff7 with result from task 899 parsl.dataflow.dflow DEBUG: Task 899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/4269cdcf-1ff1-4ae8-8384-83257574a98d_calibrate_945869_23.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/4269cdcf-1ff1-4ae8-8384-83257574a98d_calibrate_945869_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:21 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:22 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:22 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:22 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18e91ca72d1d1174b90b1f4c6861f95e with result from task 793 parsl.dataflow.dflow DEBUG: Task 793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash fafdfc5415121206f313571fb0bbae8c parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246345c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 1057 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79e05558-647a-4616-9ef6-08b20e566b16_calibrate_945868_21.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79e05558-647a-4616-9ef6-08b20e566b16_calibrate_945868_21.stderr parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/1b62231d-cf50-4958-915b-388d6f09dbe6_characterizeImage_945868_21.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/1b62231d-cf50-4958-915b-388d6f09dbe6_characterizeImage_945868_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:22 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9a4e30b97e188e6289bb32e00d3ea2e with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/52ed28fc-7c3d-49fe-8afc-8b7611f27df0_calibrate_945869_15.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/52ed28fc-7c3d-49fe-8afc-8b7611f27df0_calibrate_945869_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:23 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:23 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 204a9f9ba73566d2e732869df3c48361 with result from task 991 parsl.dataflow.dflow DEBUG: Task 991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 992 has memoization hash dc5cd0622ed6119da33e69e1470a6aca parsl.dataflow.memoization INFO: Task 992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 992 try 0 launched on executor multi with executor id 1058 parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/7cd6db84-44da-4a0e-968d-6bb7a9ff6f8f_calibrate_945757_25.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/7cd6db84-44da-4a0e-968d-6bb7a9ff6f8f_calibrate_945757_25.stderr parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a7a495bc-10ba-4ef0-95ce-0273d6f2c9cc_characterizeImage_945757_25.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/a7a495bc-10ba-4ef0-95ce-0273d6f2c9cc_characterizeImage_945757_25.stderr parsl.dataflow.dflow INFO: Task 985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c637ddc40217aa3c99cc0c7ba5a67253 with result from task 985 parsl.dataflow.dflow DEBUG: Task 985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 986 has memoization hash 6aba50da099ef949c3307e7036cb5dd7 parsl.dataflow.memoization INFO: Task 986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246228220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 986 try 0 launched on executor multi with executor id 1059 parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d7f28f17-1349-4130-8f64-00d13e60b800_calibrate_945757_28.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d7f28f17-1349-4130-8f64-00d13e60b800_calibrate_945757_28.stderr parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/4718b404-6045-4215-aee8-bd38a8817346_characterizeImage_945757_28.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/4718b404-6045-4215-aee8-bd38a8817346_characterizeImage_945757_28.stderr parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53e2e108b8b32f90668894393f315e7a with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 873f8e9a2860921b1e7cd9e61ebc3274 parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 1060 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/28c8ce88-b068-47d9-bbcf-9a95638ab36d_calibrate_945869_1.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/28c8ce88-b068-47d9-bbcf-9a95638ab36d_calibrate_945869_1.stderr parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/ca976092-8cb6-4b69-8ec7-5909a5643943_characterizeImage_945869_1.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/ca976092-8cb6-4b69-8ec7-5909a5643943_characterizeImage_945869_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:23 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:23 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:24 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7678c87b7162b97e9c240e26f01ad655 with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b5767f11-9522-4a31-b1c8-a5767bea913e_calibrate_945868_6.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b5767f11-9522-4a31-b1c8-a5767bea913e_calibrate_945868_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:24 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af9d02360ab88f1bdef4c5b0d7912350 with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/9a352527-3ac2-4dc8-ba3b-27b2098170da_calibrate_945867_10.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/9a352527-3ac2-4dc8-ba3b-27b2098170da_calibrate_945867_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:24 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 130ba1b07dce851ebdae533cf53989a1 with result from task 952 parsl.dataflow.dflow DEBUG: Task 952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 953 has memoization hash 49f4366990b5d800cf34ac9b03c1b9b9 parsl.dataflow.memoization INFO: Task 953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 1061 parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/5f1aef2c-7d7f-455c-82a2-c1f2b637ee1f_calibrate_945755_18.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/5f1aef2c-7d7f-455c-82a2-c1f2b637ee1f_calibrate_945755_18.stderr parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/2d1bcfe6-b009-42af-a161-4cbc4ccadd5e_characterizeImage_945755_18.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/2d1bcfe6-b009-42af-a161-4cbc4ccadd5e_characterizeImage_945755_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:24 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 818 try 1 failed parsl.dataflow.dflow ERROR: Task 818 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry aeb38d6db1fd7a16a96bb9771a9528a5 with result from task 818 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b9d1d390-934e-4c1e-9010-8a2e557e071e_calibrate_945868_31.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b9d1d390-934e-4c1e-9010-8a2e557e071e_calibrate_945868_31.stderr parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9488a0a58b87ff57591f6b157e0f4a62 with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash 1c53157294947fa2b62a3479f561d9a4 parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 1062 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/49f23899-34b6-4bb1-84a6-6368a7cf043c_calibrate_945867_14.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/49f23899-34b6-4bb1-84a6-6368a7cf043c_calibrate_945867_14.stderr parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/7fd7e4c5-c5d9-49e9-8392-c3b5f25a4138_characterizeImage_945867_14.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/7fd7e4c5-c5d9-49e9-8392-c3b5f25a4138_characterizeImage_945867_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:25 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:25 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:25 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a37cb82effb4b3ee4b0e4ce22353a45f with result from task 1051 parsl.dataflow.dflow DEBUG: Task 1051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash d7ea3ab2c65f9b87b5ccb002f6314229 parsl.dataflow.memoization INFO: Task 1052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 1063 parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/55844478-55b4-4394-ad3f-cf3743128ffa_calibrate_945868_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/55844478-55b4-4394-ad3f-cf3743128ffa_calibrate_945868_55.stderr parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/45e915ff-829a-403e-b624-b82f2159ff0b_characterizeImage_945868_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/45e915ff-829a-403e-b624-b82f2159ff0b_characterizeImage_945868_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:25 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:26 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:26 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 355c1040bca6a8385bff1380a7ef9bd1 with result from task 881 parsl.dataflow.dflow DEBUG: Task 881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/3817ff25-fb82-4c67-9c9b-d517fbd019b9_calibrate_945869_39.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/3817ff25-fb82-4c67-9c9b-d517fbd019b9_calibrate_945869_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:26 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:26 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39a31b3237e1a0ff0bcade57ec07184c with result from task 989 parsl.dataflow.dflow DEBUG: Task 989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7e3c9590-bf95-4316-adcc-bb21a9f9ab52_calibrate_945755_48.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/7e3c9590-bf95-4316-adcc-bb21a9f9ab52_calibrate_945755_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:27 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:27 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c11632f5499dc1c691bad509e5cac643 with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8e584c07-7629-4ab5-8710-de14d4431b9e_calibrate_945756_43.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/8e584c07-7629-4ab5-8710-de14d4431b9e_calibrate_945756_43.stderr parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83ac01ecd08085d03f3764fb26512ab9 with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c8b44883-2f53-444e-89fd-1178198620c9_calibrate_945756_34.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/c8b44883-2f53-444e-89fd-1178198620c9_calibrate_945756_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:27 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b6033948fa69de29e5ca8802d7e55f3 with result from task 1036 parsl.dataflow.dflow DEBUG: Task 1036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash d9d3431c2f8f0570629178a4f413e997 parsl.dataflow.memoization INFO: Task 1037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246089a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 1064 parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/67c14a93-abb0-4a65-b45e-0b7000470675_calibrate_945868_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/67c14a93-abb0-4a65-b45e-0b7000470675_calibrate_945868_42.stderr parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/e72b5611-0055-47de-b273-203c6208d345_characterizeImage_945868_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/e72b5611-0055-47de-b273-203c6208d345_characterizeImage_945868_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:27 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:28 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:28 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:28 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbe68efac3d066fac9c80ae36bf3e243 with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/b672ddc8-3857-47d1-953f-803a34aacce7_calibrate_945867_1.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/b672ddc8-3857-47d1-953f-803a34aacce7_calibrate_945867_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:28 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dd40bf7551e4ddfba12d09353acf113 with result from task 935 parsl.dataflow.dflow DEBUG: Task 935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/a4761586-38ff-48d7-9e23-eca5d03ac51d_calibrate_945756_6.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/a4761586-38ff-48d7-9e23-eca5d03ac51d_calibrate_945756_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:29 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 428858b8c64a928f1a238e8cc3004a86 with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c759e4a0-ead1-4dc8-b157-74725a07e8e6_calibrate_945867_33.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c759e4a0-ead1-4dc8-b157-74725a07e8e6_calibrate_945867_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:29 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:29 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d28fc721310b1dfae25638e03c16c8a with result from task 1012 parsl.dataflow.dflow DEBUG: Task 1012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash 945d7f1b4cdf8eba5c462ee7878cb18c parsl.dataflow.memoization INFO: Task 1013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246088180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 1065 parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/cd1d0ba5-4003-4dfe-97a6-694280c78cb1_calibrate_945867_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/cd1d0ba5-4003-4dfe-97a6-694280c78cb1_calibrate_945867_19.stderr parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ec6d412e-73c8-4b7c-8ee8-386f556a4afb_characterizeImage_945867_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ec6d412e-73c8-4b7c-8ee8-386f556a4afb_characterizeImage_945867_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:29 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:30 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:30 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bd7302910a5b22433e7515bb2797716 with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2e5d5288-0bf3-4eeb-8c3b-ae6ebbdbcd95_calibrate_945868_20.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2e5d5288-0bf3-4eeb-8c3b-ae6ebbdbcd95_calibrate_945868_20.stderr parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54379e0520a208cc2069d9cd2d3e5cb5 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash 5ed24ba9f9c423b1641c9dc985606e72 parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 1066 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d3f7f8ac-c906-4ebb-bc83-7030cc015953_calibrate_945755_12.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d3f7f8ac-c906-4ebb-bc83-7030cc015953_calibrate_945755_12.stderr parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/adeaca90-bd4c-4845-8619-13aa2c0abed2_characterizeImage_945755_12.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/adeaca90-bd4c-4845-8619-13aa2c0abed2_characterizeImage_945755_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:30 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef3ab0473380b10cb244dc9718abc399 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/4f99a9d2-ce67-45a8-a913-70a90c12158a_calibrate_945756_44.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/4f99a9d2-ce67-45a8-a913-70a90c12158a_calibrate_945756_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:30 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adaefc9ea30373fb2e4ff617d2539ee9 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/69de406a-2cfe-454d-a524-98bd0f3cebde_calibrate_945869_5.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/69de406a-2cfe-454d-a524-98bd0f3cebde_calibrate_945869_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:30 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8818784a5b34a7cc0b7326bb57f2a060 with result from task 1031 parsl.dataflow.dflow DEBUG: Task 1031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/3da6387d-05a0-47c6-8d07-a7eecfac4a7c_calibrate_945757_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/3da6387d-05a0-47c6-8d07-a7eecfac4a7c_calibrate_945757_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:31 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:31 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:31 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:31 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb097d84c298f786e0a7cb2483b2e3f3 with result from task 937 parsl.dataflow.dflow DEBUG: Task 937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 938 has memoization hash 072bb733492875b7c4c44142952073ba parsl.dataflow.memoization INFO: Task 938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246228ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 1067 parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/07aada8f-cb8d-45b4-a71b-6f27952d2add_calibrate_945869_43.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/07aada8f-cb8d-45b4-a71b-6f27952d2add_calibrate_945869_43.stderr parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/310b0d1b-2f32-45a4-8e12-af194b96803d_characterizeImage_945869_43.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/310b0d1b-2f32-45a4-8e12-af194b96803d_characterizeImage_945869_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:32 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21191dc2a63b000d300597eb803b9291 with result from task 811 parsl.dataflow.dflow DEBUG: Task 811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 812 has memoization hash da2293d29e4d540603424f4b2ab6d852 parsl.dataflow.memoization INFO: Task 812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246346f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 1068 parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8dcfb45a-8dc8-4688-b73c-e520bfd6d3e5_calibrate_945868_14.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8dcfb45a-8dc8-4688-b73c-e520bfd6d3e5_calibrate_945868_14.stderr parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/d2bd6043-267b-4e61-a296-80c2fbc509ba_characterizeImage_945868_14.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/d2bd6043-267b-4e61-a296-80c2fbc509ba_characterizeImage_945868_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:32 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:32 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:32 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:33 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 619f0f9310936fcb5485ee1bf5ed185b with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 91d241dd57d13c2a05fc1ca1a96bf3be parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 1069 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/9a5eba25-9c0f-4418-894d-1e8a86844831_calibrate_945756_33.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/9a5eba25-9c0f-4418-894d-1e8a86844831_calibrate_945756_33.stderr parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/48732780-b30b-4b32-8fba-d19a9a6f2ccd_characterizeImage_945756_33.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/48732780-b30b-4b32-8fba-d19a9a6f2ccd_characterizeImage_945756_33.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe84007f2565a53beedcec196d35516e with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/d82f216d-14de-4cad-b130-83f8223bd35f_calibrate_945868_17.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/d82f216d-14de-4cad-b130-83f8223bd35f_calibrate_945868_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:33 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:33 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb011fc2268f10f71ce31c2e848c29c5 with result from task 950 parsl.dataflow.dflow DEBUG: Task 950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/56731c46-9d22-4424-b9c7-09658c87eb66_calibrate_945869_21.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/56731c46-9d22-4424-b9c7-09658c87eb66_calibrate_945869_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:33 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:34 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 116 parsl.dataflow.strategy DEBUG: Executor multi has 116 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 116 parsl.dataflow.strategy DEBUG: Executor multi has 116 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baa4a222b2746fec25a532ba10592491 with result from task 920 parsl.dataflow.dflow DEBUG: Task 920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/74355852-3cd3-46ed-a6b3-77d51dbded55_calibrate_945755_44.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/74355852-3cd3-46ed-a6b3-77d51dbded55_calibrate_945755_44.stderr parsl.dataflow.dflow INFO: Task 994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36204457327ec5e4efab65fd4064baeb with result from task 994 parsl.dataflow.dflow DEBUG: Task 994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 995 has memoization hash b0c64dd832bd077e7feea27d1d5a2d76 parsl.dataflow.memoization INFO: Task 995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 1070 parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/0621298a-6b00-4a15-bb2e-fc856bb23d3c_calibrate_945755_51.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/0621298a-6b00-4a15-bb2e-fc856bb23d3c_calibrate_945755_51.stderr parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/e9b885eb-ec59-4d8e-bfd4-57a2b8ea98b3_characterizeImage_945755_51.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/e9b885eb-ec59-4d8e-bfd4-57a2b8ea98b3_characterizeImage_945755_51.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a0747e4dec5ebd83cf421551cf3c4d6 with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/46bdea0d-5577-445d-a0cb-67fffb2a508b_calibrate_945868_32.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/46bdea0d-5577-445d-a0cb-67fffb2a508b_calibrate_945868_32.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b1b948ae9dae59e8b8ade889d8f28ea with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash b935a1dbb5736082c50c7e97ebedfc04 parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246400220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 1071 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/02afb6f6-0a0b-4234-8444-b7b324554333_calibrate_945756_7.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/02afb6f6-0a0b-4234-8444-b7b324554333_calibrate_945756_7.stderr parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/2cf7ab96-403c-4e6b-b4cb-33c57bd48bef_characterizeImage_945756_7.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/2cf7ab96-403c-4e6b-b4cb-33c57bd48bef_characterizeImage_945756_7.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62e16edf017df1672226ecf66e987af1 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/500c4347-44a2-4097-8b0b-f21fea45d785_calibrate_945868_4.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/500c4347-44a2-4097-8b0b-f21fea45d785_calibrate_945868_4.stderr parsl.dataflow.dflow INFO: Task 1097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbbdf312bb5bfcf5c1915d3566edf82d with result from task 1097 parsl.dataflow.dflow DEBUG: Task 1097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/5c0b3fd9-8e29-4b9a-b660-4cdb60c0ef84_calibrate_945867_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/5c0b3fd9-8e29-4b9a-b660-4cdb60c0ef84_calibrate_945867_3.stderr parsl.dataflow.dflow INFO: Task 1046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5040f0cadc5c6ec981b77a75aec53c57 with result from task 1046 parsl.dataflow.dflow DEBUG: Task 1046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/ce58f9eb-9df0-4716-98b8-379d6941a288_calibrate_945756_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/ce58f9eb-9df0-4716-98b8-379d6941a288_calibrate_945756_16.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33113f3e3a59a438d2e8145e1258c086 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash ad10a91a0d83040f8b22f81ffc186e95 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463925c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 1072 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f636a7c1-86e2-4eb1-b303-c2a719d5f375_calibrate_945755_1.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f636a7c1-86e2-4eb1-b303-c2a719d5f375_calibrate_945755_1.stderr parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/3a9c1e61-0c0f-4e33-8cae-b1c4b86c3472_characterizeImage_945755_1.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/3a9c1e61-0c0f-4e33-8cae-b1c4b86c3472_characterizeImage_945755_1.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a527d3c0d88a6646f7a07647ecf27ee with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 85d589edabde08bd8f5626a4c7c56c27 parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463b2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 1073 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/65da99ae-6e67-4532-bbb4-0e87d8d4bf71_calibrate_945755_6.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/65da99ae-6e67-4532-bbb4-0e87d8d4bf71_calibrate_945755_6.stderr parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/874e4591-476b-4ff5-8c6e-e8519ea1a817_characterizeImage_945755_6.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/874e4591-476b-4ff5-8c6e-e8519ea1a817_characterizeImage_945755_6.stderr parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb06820246f47129852434df70c2fe94 with result from task 751 parsl.dataflow.dflow DEBUG: Task 751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 752 has memoization hash 1014f34af9ac1a958c11e5eacdeecde0 parsl.dataflow.memoization INFO: Task 752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246312840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 1074 parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f7076aa8-c124-4bb5-9e2c-136f157f9d23_calibrate_945868_18.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f7076aa8-c124-4bb5-9e2c-136f157f9d23_calibrate_945868_18.stderr parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/c8de9a78-6da7-42c0-b7bf-500ad82c1870_characterizeImage_945868_18.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/c8de9a78-6da7-42c0-b7bf-500ad82c1870_characterizeImage_945868_18.stderr parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1c9a4e12206d76c08134ddc7ddd3d8b with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash a7fbffb5d36eeeb5afe9d5e86d122f33 parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246402480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 1075 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/42f10031-6b3c-4561-89fa-00ff6b136c9f_calibrate_945867_22.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/42f10031-6b3c-4561-89fa-00ff6b136c9f_calibrate_945867_22.stderr parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/7e5f4661-d697-4084-919d-4f2c2919ffb7_characterizeImage_945867_22.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/7e5f4661-d697-4084-919d-4f2c2919ffb7_characterizeImage_945867_22.stderr parsl.dataflow.dflow INFO: Task 1018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d25570846ac3a6847d829347dc4ef3f0 with result from task 1018 parsl.dataflow.dflow DEBUG: Task 1018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash 91f5c923fdb5bd554ef3119101514f76 parsl.dataflow.memoization INFO: Task 1019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460887c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 1076 parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/45d0711e-4279-4cbd-8644-a529df22e4ff_calibrate_945867_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/45d0711e-4279-4cbd-8644-a529df22e4ff_calibrate_945867_35.stderr parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/d895e69a-5887-4611-93f2-02f0bb7c2ace_characterizeImage_945867_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/d895e69a-5887-4611-93f2-02f0bb7c2ace_characterizeImage_945867_35.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 111 parsl.dataflow.strategy DEBUG: Executor multi has 111 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.dflow INFO: Task 964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79ec8be402a45bd26f34a4ff2c53f51e with result from task 964 parsl.dataflow.dflow DEBUG: Task 964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 965 has memoization hash 64809adb8120c1df8c0a5b04e039b4fb parsl.dataflow.memoization INFO: Task 965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 965 try 0 launched on executor multi with executor id 1077 parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/a0d239a0-bcaa-4587-89b1-de89c80a7046_calibrate_945757_24.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/a0d239a0-bcaa-4587-89b1-de89c80a7046_calibrate_945757_24.stderr parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/966446c6-7a4d-41b2-b353-7e0c9421f0c5_characterizeImage_945757_24.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/966446c6-7a4d-41b2-b353-7e0c9421f0c5_characterizeImage_945757_24.stderr parsl.dataflow.dflow INFO: Task 1063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9917cb3d5a2d05e5eca33b80fb9cd4a with result from task 1063 parsl.dataflow.dflow DEBUG: Task 1063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1064 has memoization hash 7c21dd59e3017abb9deb240a32129364 parsl.dataflow.memoization INFO: Task 1064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1064 try 0 launched on executor multi with executor id 1078 parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/ef06335b-9d3a-4659-b18d-e3a87f34bc8b_calibrate_945869_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/ef06335b-9d3a-4659-b18d-e3a87f34bc8b_calibrate_945869_42.stderr parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/3eae7396-a860-43be-9c75-20db509ecc36_characterizeImage_945869_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/3eae7396-a860-43be-9c75-20db509ecc36_characterizeImage_945869_42.stderr parsl.dataflow.dflow INFO: Task 1049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15fa4ba761f3315d637e55f6eb47982c with result from task 1049 parsl.dataflow.dflow DEBUG: Task 1049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/3bdc4835-6352-48a2-b9aa-7cb492c3784e_calibrate_945756_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/3bdc4835-6352-48a2-b9aa-7cb492c3784e_calibrate_945756_4.stderr parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ca82012e96ec8f372ac91f2e771f4e0 with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/ec6d8339-706b-46c4-8bb5-5bf3e47be1ea_calibrate_945757_37.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/ec6d8339-706b-46c4-8bb5-5bf3e47be1ea_calibrate_945757_37.stderr parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0fb53ebdb45cef709571cc4ff6b10d2 with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/d46f222c-4de1-4608-a13b-c75f005e7257_calibrate_945868_62.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/d46f222c-4de1-4608-a13b-c75f005e7257_calibrate_945868_62.stderr parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f26c5b48f859c8014b77a6444ab58f47 with result from task 797 parsl.dataflow.dflow DEBUG: Task 797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/85de58a6-4deb-406e-8959-d16ff68f750d_calibrate_945869_7.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/85de58a6-4deb-406e-8959-d16ff68f750d_calibrate_945869_7.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a55eb2d19340a19030c56918fbe92287 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/108d6122-57ff-4fc3-bd11-96e1983822f0_calibrate_945868_60.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/108d6122-57ff-4fc3-bd11-96e1983822f0_calibrate_945868_60.stderr parsl.dataflow.dflow INFO: Task 982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80357df3c28238a48776038387ef2db7 with result from task 982 parsl.dataflow.dflow DEBUG: Task 982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 983 has memoization hash 2a005f3c313060d01d5f3965ba1654ff parsl.dataflow.memoization INFO: Task 983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 1079 parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/281c5728-24f4-415b-8aa8-0b1b32c9b525_calibrate_945869_38.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/281c5728-24f4-415b-8aa8-0b1b32c9b525_calibrate_945869_38.stderr parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/893ec485-1dd1-4ee0-984d-6e5596cb027f_characterizeImage_945869_38.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/893ec485-1dd1-4ee0-984d-6e5596cb027f_characterizeImage_945869_38.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f167d103250465c67334955dc47dbf0c with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/55408845-e6d5-4b76-b2d9-981be02dd657_calibrate_945867_30.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/55408845-e6d5-4b76-b2d9-981be02dd657_calibrate_945867_30.stderr 2024-03-13 18:38:44 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:38:44 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:44 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8531d9692a4349e1d6ecee51649c51bb with result from task 851 parsl.dataflow.dflow DEBUG: Task 851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0922e95e-3f5c-4520-bef7-16b6c16eede8_calibrate_945869_58.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0922e95e-3f5c-4520-bef7-16b6c16eede8_calibrate_945869_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:44 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:44 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:45 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:45 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 591b371a4425ea2077ad027b5b2827b9 with result from task 767 parsl.dataflow.dflow DEBUG: Task 767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f80edba8-0bc0-44d5-9957-d11f654800d6_calibrate_945755_25.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f80edba8-0bc0-44d5-9957-d11f654800d6_calibrate_945755_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:45 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:45 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:46 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:46 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 1010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6558664b2fbd1473ae1525621fe4433a with result from task 1010 parsl.dataflow.dflow DEBUG: Task 1010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/fd2b3ecb-825c-418e-be28-fd9d58953737_calibrate_945756_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/fd2b3ecb-825c-418e-be28-fd9d58953737_calibrate_945756_25.stderr parsl.dataflow.dflow INFO: Task 931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29e90f41472b81dd37ea84c625c1260b with result from task 931 parsl.dataflow.dflow DEBUG: Task 931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 932 has memoization hash 4010fa711c89b8db84773f24a8f5cb2b parsl.dataflow.memoization INFO: Task 932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246229120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 1080 parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/566395f7-96bc-4d6b-8af5-e44c7b39144e_calibrate_945755_13.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/566395f7-96bc-4d6b-8af5-e44c7b39144e_calibrate_945755_13.stderr parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/1f5b32d1-f002-4854-906f-f9337c678523_characterizeImage_945755_13.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/1f5b32d1-f002-4854-906f-f9337c678523_characterizeImage_945755_13.stderr parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f44056c52c81499b9fa3a61cdbe3094 with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 906ff7400ddd1ec425c998bd5e3068f8 parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 1081 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/c5cdef27-1453-496d-87b4-20e77c20c559_calibrate_945757_27.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/c5cdef27-1453-496d-87b4-20e77c20c559_calibrate_945757_27.stderr parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/13ee89be-7195-4ef4-8c17-418966cc4353_characterizeImage_945757_27.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/13ee89be-7195-4ef4-8c17-418966cc4353_characterizeImage_945757_27.stderr parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ba904eaecc5e2aa7e45218dbe4e8497 with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 39c09ff284d52ecca4d0098a3800bbb4 parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246402d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 1082 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/c518af20-e80e-4122-9b67-d2fe88d9b0ac_calibrate_945755_10.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/c518af20-e80e-4122-9b67-d2fe88d9b0ac_calibrate_945755_10.stderr parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/86c57a26-7a15-46ed-8874-e8b53f9d2328_characterizeImage_945755_10.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/86c57a26-7a15-46ed-8874-e8b53f9d2328_characterizeImage_945755_10.stderr parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0977e5a6965281a564153c961e7c2cda with result from task 922 parsl.dataflow.dflow DEBUG: Task 922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 923 has memoization hash 17424445ca3ebffedc2045037efeac0b parsl.dataflow.memoization INFO: Task 923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 1083 parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/b8588fa2-3605-48c6-ad60-50c5226e3595_calibrate_945756_3.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/b8588fa2-3605-48c6-ad60-50c5226e3595_calibrate_945756_3.stderr parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6759b505-bf94-47cd-8bac-678eea613eb7_characterizeImage_945756_3.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/6759b505-bf94-47cd-8bac-678eea613eb7_characterizeImage_945756_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:46 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79b747af09acee318c5b44337b86297a with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 51c32163072d090133db69643d3e51e2 parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 1084 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/ff0d1bcc-4431-46b7-8bd4-a1ba3e6ef25f_calibrate_945757_29.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/ff0d1bcc-4431-46b7-8bd4-a1ba3e6ef25f_calibrate_945757_29.stderr parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/18a41296-398c-449e-b7a7-d00591d488fd_characterizeImage_945757_29.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/18a41296-398c-449e-b7a7-d00591d488fd_characterizeImage_945757_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:46 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:47 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:47 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:47 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d61b43c4fece7ad1a65eb75bed7fe5a5 with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f01de854-a0ad-4055-b58a-388488cf4f5d_calibrate_945868_10.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f01de854-a0ad-4055-b58a-388488cf4f5d_calibrate_945868_10.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 605 try 1 failed parsl.dataflow.dflow ERROR: Task 605 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c5f2e535a1a1a7141b8baa906d73fbe3 with result from task 605 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/22d1d061-a587-4cc3-8506-6319e7ebe162_calibrate_945869_11.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/22d1d061-a587-4cc3-8506-6319e7ebe162_calibrate_945869_11.stderr parsl.dataflow.dflow INFO: Task 997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 313389040ad634ca8f35e7fc282bfb3f with result from task 997 parsl.dataflow.dflow DEBUG: Task 997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 998 has memoization hash 0cd4ce5f76707e903a1112cd050724aa parsl.dataflow.memoization INFO: Task 998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924622b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 1085 parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4a2ba8dc-0bbf-4e49-818c-3905483a2b36_calibrate_945757_60.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4a2ba8dc-0bbf-4e49-818c-3905483a2b36_calibrate_945757_60.stderr parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/ffc29455-e1ae-4685-bc0e-dde333770849_characterizeImage_945757_60.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/ffc29455-e1ae-4685-bc0e-dde333770849_characterizeImage_945757_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:47 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:48 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49f4366990b5d800cf34ac9b03c1b9b9 with result from task 953 parsl.dataflow.dflow DEBUG: Task 953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/5f1aef2c-7d7f-455c-82a2-c1f2b637ee1f_calibrate_945755_18.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/5f1aef2c-7d7f-455c-82a2-c1f2b637ee1f_calibrate_945755_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:48 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:48 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:48 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:49 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:49 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c25fa1655e5af17d59719ef602d1f833 with result from task 785 parsl.dataflow.dflow DEBUG: Task 785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/4bde8942-bc7b-4fa8-90c6-7d7c3cbc92da_calibrate_945869_28.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/4bde8942-bc7b-4fa8-90c6-7d7c3cbc92da_calibrate_945869_28.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ccc05a6be84028ed87e817dc8866a98 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/964243a1-3945-426a-a763-2d80eafe2027_calibrate_945867_29.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/964243a1-3945-426a-a763-2d80eafe2027_calibrate_945867_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:49 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51786b64b84ab15c7bbe90ae90f751b2 with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/0a90b0b4-5371-4110-a8a5-fabaec267249_calibrate_945867_12.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/0a90b0b4-5371-4110-a8a5-fabaec267249_calibrate_945867_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:49 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:50 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:50 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:50 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb23b14992a0431f6fe77ce967f3e4fc with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash bed557de8ff9dd372e04beda15fbb647 parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246400360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 1086 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1effc39e-f9e2-4f24-be3d-d58004c22278_calibrate_945869_35.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1effc39e-f9e2-4f24-be3d-d58004c22278_calibrate_945869_35.stderr parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/5e488abd-6d73-4d04-b66a-3bc712c9f5ac_characterizeImage_945869_35.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/5e488abd-6d73-4d04-b66a-3bc712c9f5ac_characterizeImage_945869_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:50 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:51 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:51 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:51 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e73f5a8bbff7714c19f66689e6dd0351 with result from task 758 parsl.dataflow.dflow DEBUG: Task 758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/3b7eb95b-8476-4e6b-9de5-e26593f4958c_calibrate_945867_47.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/3b7eb95b-8476-4e6b-9de5-e26593f4958c_calibrate_945867_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:51 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cf898c3a4d7dc898109a6b18ea064df with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/2f8ea3ea-82df-4578-9632-2be297842c8a_calibrate_945757_21.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/2f8ea3ea-82df-4578-9632-2be297842c8a_calibrate_945757_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:51 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc5cd0622ed6119da33e69e1470a6aca with result from task 992 parsl.dataflow.dflow DEBUG: Task 992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/7cd6db84-44da-4a0e-968d-6bb7a9ff6f8f_calibrate_945757_25.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/7cd6db84-44da-4a0e-968d-6bb7a9ff6f8f_calibrate_945757_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:52 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:52 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 895b872d89f0d3e887f9d036ab910f7a with result from task 890 parsl.dataflow.dflow DEBUG: Task 890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f40b16bb-4c14-4f49-9e12-3c100790dbf5_calibrate_945868_45.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f40b16bb-4c14-4f49-9e12-3c100790dbf5_calibrate_945868_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:52 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:52 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 1039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c5813b3bc0be26948594abe53e6d4dc with result from task 1039 parsl.dataflow.dflow DEBUG: Task 1039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash b9c257e53b0789e8223377aa1817f7b8 parsl.dataflow.memoization INFO: Task 1040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246089da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 1087 parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/6f1c9aee-33f4-4773-aaf8-7cbf316794a9_calibrate_945867_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/6f1c9aee-33f4-4773-aaf8-7cbf316794a9_calibrate_945867_52.stderr parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/a28c361b-ba1c-461c-a8b9-5b791fe4c479_characterizeImage_945867_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/a28c361b-ba1c-461c-a8b9-5b791fe4c479_characterizeImage_945867_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:53 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:53 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:53 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:54 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:54 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:54 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:54 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:55 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa00da1d1a651c9a03a52ff6c0625635 with result from task 979 parsl.dataflow.dflow DEBUG: Task 979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 980 has memoization hash 13744e01d251aadcf9387d768f6eaf32 parsl.dataflow.memoization INFO: Task 980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 1088 parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/5a566a9b-f70a-4dd5-8f27-2347c7668f05_calibrate_945756_58.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/5a566a9b-f70a-4dd5-8f27-2347c7668f05_calibrate_945756_58.stderr parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/422061cf-a9c5-4de0-aa31-1d92caabccc6_characterizeImage_945756_58.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/422061cf-a9c5-4de0-aa31-1d92caabccc6_characterizeImage_945756_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:55 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:55 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:55 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aba50da099ef949c3307e7036cb5dd7 with result from task 986 parsl.dataflow.dflow DEBUG: Task 986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d7f28f17-1349-4130-8f64-00d13e60b800_calibrate_945757_28.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/d7f28f17-1349-4130-8f64-00d13e60b800_calibrate_945757_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:56 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:56 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:56 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b17f571d8129f6a7c1bbaf1ac8af3e9f with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/7be7d14d-3921-4368-af4b-a5e7a4cd7f6a_calibrate_945868_26.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/7be7d14d-3921-4368-af4b-a5e7a4cd7f6a_calibrate_945868_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:56 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:57 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84a8ee38703cf23ce76c0161af0b315b with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/8e106d20-2da4-4238-8eed-f3f7d42430fa_calibrate_945867_34.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/8e106d20-2da4-4238-8eed-f3f7d42430fa_calibrate_945867_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:57 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ed24ba9f9c423b1641c9dc985606e72 with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d3f7f8ac-c906-4ebb-bc83-7030cc015953_calibrate_945755_12.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/d3f7f8ac-c906-4ebb-bc83-7030cc015953_calibrate_945755_12.stderr parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdae1e5cbfac39c13c3bc4afce8415e2 with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash b0b41ab5e9acf1572ccb4dd4e8c334fd parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246400cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:57 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 1089 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/4e3666d4-b01a-4d41-a065-ade98f63d940_calibrate_945868_33.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/4e3666d4-b01a-4d41-a065-ade98f63d940_calibrate_945868_33.stderr parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a6138588-6fff-449f-9079-639b0c79a6cb_characterizeImage_945868_33.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a6138588-6fff-449f-9079-639b0c79a6cb_characterizeImage_945868_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:57 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:58 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:58 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:58 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:58 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:59 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:59 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:59 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:38:59 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:00 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:00 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:00 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:00 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5 try 1 failed parsl.dataflow.dflow ERROR: Task 5 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 0d76ac2f13d8970266d0abc4a3dc6c07 with result from task 5 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/95678e94-3e5f-4805-804a-55a1881d9410_calibrate_945868_11.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/95678e94-3e5f-4805-804a-55a1881d9410_calibrate_945868_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:01 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3733c36746d057a112c041da3dc61ba with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/2c49a2ce-729b-4e1d-8c1b-46795ce508bd_calibrate_945869_40.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/2c49a2ce-729b-4e1d-8c1b-46795ce508bd_calibrate_945869_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:01 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:01 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:01 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:02 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:02 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e89c8bc9c33098ebcaca8bc54564b3d4 with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/9ee28858-35bc-4268-8505-3581e5f87efb_calibrate_945868_47.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/9ee28858-35bc-4268-8505-3581e5f87efb_calibrate_945868_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:02 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 765d37a9e712c334493c8bbf6ee4b5fe with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/0277ca91-96b2-4093-a122-8ba2708b68bd_calibrate_945867_18.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/0277ca91-96b2-4093-a122-8ba2708b68bd_calibrate_945867_18.stderr parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce0f7ffa88e68d17148e3440655f0927 with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash 19516bf21694e95df5a619b434167531 parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492464039c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 1090 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/d2b0b567-6e54-43bd-afc5-a2579ea4fad2_calibrate_945869_16.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/d2b0b567-6e54-43bd-afc5-a2579ea4fad2_calibrate_945869_16.stderr parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bbcb88ca-1d01-4b2e-9050-e5d20dc0de14_characterizeImage_945869_16.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bbcb88ca-1d01-4b2e-9050-e5d20dc0de14_characterizeImage_945869_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:02 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:03 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:03 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:03 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b247f85ebd3b629feea7fbc751b5d17 with result from task 1042 parsl.dataflow.dflow DEBUG: Task 1042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1043 has memoization hash 13eea29505fdd328b547eeac054c3b64 parsl.dataflow.memoization INFO: Task 1043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924608a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1043 try 0 launched on executor multi with executor id 1091 parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2fe8df07-a9bd-477d-b8ea-4bef5f20582c_calibrate_945868_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2fe8df07-a9bd-477d-b8ea-4bef5f20582c_calibrate_945868_30.stderr parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0ee5ae70-4210-4084-9505-28ad3d6f9a1e_characterizeImage_945868_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0ee5ae70-4210-4084-9505-28ad3d6f9a1e_characterizeImage_945868_30.stderr parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd43db89902db3ea11d8591ab9f47f1d with result from task 749 parsl.dataflow.dflow DEBUG: Task 749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c3456b46-95a1-4e62-b746-5763bcb5e779_calibrate_945867_53.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c3456b46-95a1-4e62-b746-5763bcb5e779_calibrate_945867_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:03 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:04 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:04 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:04 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:04 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:05 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:05 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:05 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:05 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:06 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc45a80aff1728bca76455cca2a09be6 with result from task 1079 parsl.dataflow.dflow DEBUG: Task 1079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c99ee9b1-e7d9-4f83-96e3-f08da9e3bc43_calibrate_945867_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/c99ee9b1-e7d9-4f83-96e3-f08da9e3bc43_calibrate_945867_51.stderr parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b935a1dbb5736082c50c7e97ebedfc04 with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/02afb6f6-0a0b-4234-8444-b7b324554333_calibrate_945756_7.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/02afb6f6-0a0b-4234-8444-b7b324554333_calibrate_945756_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:06 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:06 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:06 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 81 parsl.dataflow.strategy DEBUG: Executor multi has 81 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 81 parsl.dataflow.strategy DEBUG: Executor multi has 81 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91d241dd57d13c2a05fc1ca1a96bf3be with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/9a5eba25-9c0f-4418-894d-1e8a86844831_calibrate_945756_33.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/9a5eba25-9c0f-4418-894d-1e8a86844831_calibrate_945756_33.stderr parsl.dataflow.dflow INFO: Task 1073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 017c06f8ab1761f4892b1c81e89d966a with result from task 1073 parsl.dataflow.dflow DEBUG: Task 1073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/18cfcdc5-8abb-4366-b837-259bf9cbd6ef_calibrate_945867_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/18cfcdc5-8abb-4366-b837-259bf9cbd6ef_calibrate_945867_4.stderr parsl.dataflow.dflow INFO: Task 976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2f71bfd17240aec0c94188eacac9f18 with result from task 976 parsl.dataflow.dflow DEBUG: Task 976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 977 has memoization hash 52616753b5069b8b3feab235c22722cf parsl.dataflow.memoization INFO: Task 977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 1092 parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/cb833780-3b41-4d24-b6fa-8d63c277c642_calibrate_945869_17.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/cb833780-3b41-4d24-b6fa-8d63c277c642_calibrate_945869_17.stderr parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/7243d618-f8f2-42ed-92bf-4be4f72fea3b_characterizeImage_945869_17.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/7243d618-f8f2-42ed-92bf-4be4f72fea3b_characterizeImage_945869_17.stderr parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 094fd1c844f098aeb25ead10a42e889a with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/bf5fee1e-4ad5-4bff-9f8a-52f7bc619d00_calibrate_945869_6.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/bf5fee1e-4ad5-4bff-9f8a-52f7bc619d00_calibrate_945869_6.stderr parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e409fdb5335f0b8e22ae27c57cf62d5 with result from task 593 parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/33bffe38-41e4-410a-a8c9-68efa03ae3bd_calibrate_945869_45.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/33bffe38-41e4-410a-a8c9-68efa03ae3bd_calibrate_945869_45.stderr parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fb5d46cf2b78df01bd2bf282c5d838e with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/979bf86a-4002-4d08-bd69-14a4b46c5f89_calibrate_945756_24.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/979bf86a-4002-4d08-bd69-14a4b46c5f89_calibrate_945756_24.stderr parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48fc672a5f1c88044e2476b4259ed183 with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 6d63118bed40363ab6499eae880fa3c3 parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463d76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 1093 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/67e79a2e-ea12-4715-975b-189caf86b1a6_calibrate_945867_43.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/67e79a2e-ea12-4715-975b-189caf86b1a6_calibrate_945867_43.stderr parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/247d964f-0c0c-43b4-b9e2-43a8068c5b2f_characterizeImage_945867_43.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/247d964f-0c0c-43b4-b9e2-43a8068c5b2f_characterizeImage_945867_43.stderr parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e1837ac7b963c2ef2d93b608d24392c with result from task 788 parsl.dataflow.dflow DEBUG: Task 788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/d6f99150-b0e9-4c14-8301-00558827dcc9_calibrate_945869_44.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/d6f99150-b0e9-4c14-8301-00558827dcc9_calibrate_945869_44.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bba0523eaafa9be7c6f3519a5348fca with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash b830b58f81ed859cf01ba5c52a9971ca parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246390fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 1094 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/804ae62a-9ac4-4518-8c14-6e23142a23cf_calibrate_945869_36.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/804ae62a-9ac4-4518-8c14-6e23142a23cf_calibrate_945869_36.stderr parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bd4b635b-1f03-4133-a660-8008a7656dff_characterizeImage_945869_36.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/bd4b635b-1f03-4133-a660-8008a7656dff_characterizeImage_945869_36.stderr parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdc793b7250c18b6f1e7439f5b4c9ea3 with result from task 863 parsl.dataflow.dflow DEBUG: Task 863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/ac9f7885-1e27-4437-b46b-7f29a203c4d1_calibrate_945867_28.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/ac9f7885-1e27-4437-b46b-7f29a203c4d1_calibrate_945867_28.stderr parsl.dataflow.dflow INFO: Task 995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0c64dd832bd077e7feea27d1d5a2d76 with result from task 995 parsl.dataflow.dflow DEBUG: Task 995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/0621298a-6b00-4a15-bb2e-fc856bb23d3c_calibrate_945755_51.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/0621298a-6b00-4a15-bb2e-fc856bb23d3c_calibrate_945755_51.stderr parsl.dataflow.dflow INFO: Task 932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4010fa711c89b8db84773f24a8f5cb2b with result from task 932 parsl.dataflow.dflow DEBUG: Task 932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/566395f7-96bc-4d6b-8af5-e44c7b39144e_calibrate_945755_13.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/566395f7-96bc-4d6b-8af5-e44c7b39144e_calibrate_945755_13.stderr parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b467daf806e38fbdb475b83f1ca5732 with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/25091d7d-6ade-48d5-ac04-c82b191d82f4_calibrate_945869_22.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/25091d7d-6ade-48d5-ac04-c82b191d82f4_calibrate_945869_22.stderr parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 945696b37367f484ae3a875653966480 with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/f1a31a94-24f6-43ad-ada1-ac08f1b54d9f_calibrate_945869_52.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/f1a31a94-24f6-43ad-ada1-ac08f1b54d9f_calibrate_945869_52.stderr parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a24f5dda747590be9d7282714b2e9f57 with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash 8cd8fad411e89151602c67a6241ebd9d parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 1095 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/a1570326-e40c-4661-a5bd-708cc8357d53_calibrate_945867_32.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/a1570326-e40c-4661-a5bd-708cc8357d53_calibrate_945867_32.stderr parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/57ff429e-d686-4475-95eb-c88b72f967bc_characterizeImage_945867_32.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/57ff429e-d686-4475-95eb-c88b72f967bc_characterizeImage_945867_32.stderr parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 906ff7400ddd1ec425c998bd5e3068f8 with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/c5cdef27-1453-496d-87b4-20e77c20c559_calibrate_945757_27.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/c5cdef27-1453-496d-87b4-20e77c20c559_calibrate_945757_27.stderr parsl.dataflow.dflow INFO: Task 1070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 414004bbacf4db0d425799df062bb553 with result from task 1070 parsl.dataflow.dflow DEBUG: Task 1070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ec6d2474-bd91-47ee-80ea-140d0e6daea8_calibrate_945868_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ec6d2474-bd91-47ee-80ea-140d0e6daea8_calibrate_945868_59.stderr parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39c09ff284d52ecca4d0098a3800bbb4 with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/c518af20-e80e-4122-9b67-d2fe88d9b0ac_calibrate_945755_10.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/c518af20-e80e-4122-9b67-d2fe88d9b0ac_calibrate_945755_10.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 67 parsl.dataflow.strategy DEBUG: Executor multi has 67 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5ac7bb52eff722068655aa6c8e07fcb with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/7bfc58b3-2563-4df9-ac7d-3492ec86ad4e_calibrate_945869_29.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/7bfc58b3-2563-4df9-ac7d-3492ec86ad4e_calibrate_945869_29.stderr parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc6cea0faccc541077b923a678558166 with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/9fffcef9-59fa-4217-a21c-1951e8221037_calibrate_945867_59.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/9fffcef9-59fa-4217-a21c-1951e8221037_calibrate_945867_59.stderr parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6579d1565a2969544d97367c86325b35 with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash 3724f3d6ed9f7ff87ea78fabfa18d233 parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463118a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 1096 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/392f6c4a-a9c8-46c1-bd49-c4f1a9ec5f69_calibrate_945869_50.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/392f6c4a-a9c8-46c1-bd49-c4f1a9ec5f69_calibrate_945869_50.stderr parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1375f4dc-3e17-4d19-b71b-e0ae572d84af_characterizeImage_945869_50.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/1375f4dc-3e17-4d19-b71b-e0ae572d84af_characterizeImage_945869_50.stderr parsl.dataflow.dflow INFO: Task 928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a751ec6bc8fafd88ebe9d89b186b2427 with result from task 928 parsl.dataflow.dflow DEBUG: Task 928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 929 has memoization hash 6f240812681d6893ce7e38882e126c13 parsl.dataflow.memoization INFO: Task 929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246229440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 1097 parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/bbe12b80-0328-47e2-bc8a-304f037e6608_calibrate_945757_44.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/bbe12b80-0328-47e2-bc8a-304f037e6608_calibrate_945757_44.stderr parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/3b9d20fc-d300-45bf-800c-7986868df61f_characterizeImage_945757_44.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/3b9d20fc-d300-45bf-800c-7986868df61f_characterizeImage_945757_44.stderr parsl.dataflow.dflow INFO: Task 1007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f031925461ed928d33434f9a0eed2f8 with result from task 1007 parsl.dataflow.dflow DEBUG: Task 1007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/927284b5-14f4-476b-80e9-e205145922aa_calibrate_945869_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/927284b5-14f4-476b-80e9-e205145922aa_calibrate_945869_59.stderr parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22ff4e8062edaa3dbf15ed768f2152a9 with result from task 925 parsl.dataflow.dflow DEBUG: Task 925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 926 has memoization hash 408e11f901f1d8fc6a40f71e268e61bd parsl.dataflow.memoization INFO: Task 926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246229580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 1098 parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/07f39342-4cca-40b5-8efd-ab0d677303a1_calibrate_945868_28.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/07f39342-4cca-40b5-8efd-ab0d677303a1_calibrate_945868_28.stderr parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/7927cc51-8883-4440-9dcc-db626c6eb77e_characterizeImage_945868_28.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/7927cc51-8883-4440-9dcc-db626c6eb77e_characterizeImage_945868_28.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28af6fe178a6a17d846e584d907fee89 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f91db016-e9b0-41b4-b7bd-da8a5a08293e_calibrate_945867_7.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f91db016-e9b0-41b4-b7bd-da8a5a08293e_calibrate_945867_7.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad10a91a0d83040f8b22f81ffc186e95 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f636a7c1-86e2-4eb1-b303-c2a719d5f375_calibrate_945755_1.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/f636a7c1-86e2-4eb1-b303-c2a719d5f375_calibrate_945755_1.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c53157294947fa2b62a3479f561d9a4 with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/49f23899-34b6-4bb1-84a6-6368a7cf043c_calibrate_945867_14.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/49f23899-34b6-4bb1-84a6-6368a7cf043c_calibrate_945867_14.stderr parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57515d98dfe851d64f39dbaea132c94b with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/34487c5c-03c5-4089-8dd7-c38b22e515c6_calibrate_945869_12.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/34487c5c-03c5-4089-8dd7-c38b22e515c6_calibrate_945869_12.stderr parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17424445ca3ebffedc2045037efeac0b with result from task 923 parsl.dataflow.dflow DEBUG: Task 923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/b8588fa2-3605-48c6-ad60-50c5226e3595_calibrate_945756_3.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/b8588fa2-3605-48c6-ad60-50c5226e3595_calibrate_945756_3.stderr parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51c32163072d090133db69643d3e51e2 with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/ff0d1bcc-4431-46b7-8bd4-a1ba3e6ef25f_calibrate_945757_29.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/ff0d1bcc-4431-46b7-8bd4-a1ba3e6ef25f_calibrate_945757_29.stderr parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a40d57de6920d5eb3b038113b02bbb14 with result from task 869 parsl.dataflow.dflow DEBUG: Task 869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1a91f50a-7cf4-4ee2-8d84-d2d79cbb9e3a_calibrate_945869_19.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1a91f50a-7cf4-4ee2-8d84-d2d79cbb9e3a_calibrate_945869_19.stderr 2024-03-13 18:39:17 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:39:17 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:17 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:17 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:17 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:18 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:18 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:18 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:18 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:19 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:19 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:19 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a85a16c49066520d3da3d1f7c1ac3182 with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash abd4499b9375b213e1b4edb9b0aae3ac parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 1099 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0f595126-fb35-44ad-92f4-bd00ebc681c8_calibrate_945869_49.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0f595126-fb35-44ad-92f4-bd00ebc681c8_calibrate_945869_49.stderr parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2d027c1f-06d0-4a7e-a084-6d4c950bd09d_characterizeImage_945869_49.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/2d027c1f-06d0-4a7e-a084-6d4c950bd09d_characterizeImage_945869_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:19 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:20 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64809adb8120c1df8c0a5b04e039b4fb with result from task 965 parsl.dataflow.dflow DEBUG: Task 965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/a0d239a0-bcaa-4587-89b1-de89c80a7046_calibrate_945757_24.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/a0d239a0-bcaa-4587-89b1-de89c80a7046_calibrate_945757_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:20 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:20 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 1061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cc143e020b95fc4d64bda7d028ace33 with result from task 1061 parsl.dataflow.dflow DEBUG: Task 1061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/887fad17-a8d2-4f7b-b78f-466353bf3aca_calibrate_945867_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/887fad17-a8d2-4f7b-b78f-466353bf3aca_calibrate_945867_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:20 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:21 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:21 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:21 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:21 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:22 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd176b14c30e1fb08f3a4c31f9afd614 with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash 007dcf03393a78f3b3257c6835121679 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246400680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 1100 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ea21a4b5-a1b4-4914-bf9c-bf63c99f8ad5_calibrate_945868_1.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ea21a4b5-a1b4-4914-bf9c-bf63c99f8ad5_calibrate_945868_1.stderr parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/3ef85d6b-ec2b-4c6b-bdc3-b2ac816e57ed_characterizeImage_945868_1.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/3ef85d6b-ec2b-4c6b-bdc3-b2ac816e57ed_characterizeImage_945868_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:22 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6532e09303d6190c3fced0dc4f6c85e1 with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash 7255e7163e603699c90a32cc120b3c56 parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246311bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:22 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 1101 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/a3bc32e3-91b4-456a-8623-43a8936b0e08_calibrate_945867_16.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/a3bc32e3-91b4-456a-8623-43a8936b0e08_calibrate_945867_16.stderr parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ac738be0-ef18-4266-a784-94c24406f72c_characterizeImage_945867_16.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/ac738be0-ef18-4266-a784-94c24406f72c_characterizeImage_945867_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:22 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:23 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:23 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:23 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:23 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:24 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e49299354b9f14ec5ec5d686d31a5ec with result from task 916 parsl.dataflow.dflow DEBUG: Task 916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 917 has memoization hash 40bcf869020f97601c373f01afc7573d parsl.dataflow.memoization INFO: Task 917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 1102 parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/32004859-a484-4622-9f42-1175684a24ce_calibrate_945867_23.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/32004859-a484-4622-9f42-1175684a24ce_calibrate_945867_23.stderr parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4998cc52-2253-47ff-bbcf-645062e9b75c_characterizeImage_945867_23.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/4998cc52-2253-47ff-bbcf-645062e9b75c_characterizeImage_945867_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:24 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:24 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 962acdb341ab5d825bd182abea32fd38 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8cfc81f1-8e39-49da-82f1-f45c19acdac4_calibrate_945868_29.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8cfc81f1-8e39-49da-82f1-f45c19acdac4_calibrate_945868_29.stderr parsl.dataflow.dflow INFO: Task 1022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 586f571b5c65247b8d329cd5a814d82c with result from task 1022 parsl.dataflow.dflow DEBUG: Task 1022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/e791ca8f-5a1a-41ba-bb10-4c2101541e0d_calibrate_945868_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/e791ca8f-5a1a-41ba-bb10-4c2101541e0d_calibrate_945868_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:24 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:25 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:25 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:25 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:25 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92ca9d8a864a32d515e8e27267e4a287 with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash b9547075dd1d3de1ec4dc8c533510593 parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924642eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 1103 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/8d59e95d-e0f4-4043-927e-48380c56a18e_calibrate_945757_33.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/8d59e95d-e0f4-4043-927e-48380c56a18e_calibrate_945757_33.stderr parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/5317748c-b718-4dc7-acfc-e900da9b12d1_characterizeImage_945757_33.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945757/5317748c-b718-4dc7-acfc-e900da9b12d1_characterizeImage_945757_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:26 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85d589edabde08bd8f5626a4c7c56c27 with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/65da99ae-6e67-4532-bbb4-0e87d8d4bf71_calibrate_945755_6.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/65da99ae-6e67-4532-bbb4-0e87d8d4bf71_calibrate_945755_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:26 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:26 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:27 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d61ddcffdaf1483d8fd5ebd2e3d1f9ac with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/cafd6624-5ae5-417a-b9d1-42f35222d81d_calibrate_945868_7.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/cafd6624-5ae5-417a-b9d1-42f35222d81d_calibrate_945868_7.stderr parsl.dataflow.dflow INFO: Task 980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13744e01d251aadcf9387d768f6eaf32 with result from task 980 parsl.dataflow.dflow DEBUG: Task 980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/5a566a9b-f70a-4dd5-8f27-2347c7668f05_calibrate_945756_58.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/5a566a9b-f70a-4dd5-8f27-2347c7668f05_calibrate_945756_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:27 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 1013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 945d7f1b4cdf8eba5c462ee7878cb18c with result from task 1013 parsl.dataflow.dflow DEBUG: Task 1013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/cd1d0ba5-4003-4dfe-97a6-694280c78cb1_calibrate_945867_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/cd1d0ba5-4003-4dfe-97a6-694280c78cb1_calibrate_945867_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:27 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:27 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 1037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9d3431c2f8f0570629178a4f413e997 with result from task 1037 parsl.dataflow.dflow DEBUG: Task 1037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/67c14a93-abb0-4a65-b45e-0b7000470675_calibrate_945868_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/67c14a93-abb0-4a65-b45e-0b7000470675_calibrate_945868_42.stderr parsl.dataflow.dflow INFO: Task 998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cd4ce5f76707e903a1112cd050724aa with result from task 998 parsl.dataflow.dflow DEBUG: Task 998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4a2ba8dc-0bbf-4e49-818c-3905483a2b36_calibrate_945757_60.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/4a2ba8dc-0bbf-4e49-818c-3905483a2b36_calibrate_945757_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:28 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:28 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 873f8e9a2860921b1e7cd9e61ebc3274 with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/28c8ce88-b068-47d9-bbcf-9a95638ab36d_calibrate_945869_1.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/28c8ce88-b068-47d9-bbcf-9a95638ab36d_calibrate_945869_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:28 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:29 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1014f34af9ac1a958c11e5eacdeecde0 with result from task 752 parsl.dataflow.dflow DEBUG: Task 752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f7076aa8-c124-4bb5-9e2c-136f157f9d23_calibrate_945868_18.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/f7076aa8-c124-4bb5-9e2c-136f157f9d23_calibrate_945868_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:29 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:29 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:29 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 135758735141bfe932429ed57649751b with result from task 884 parsl.dataflow.dflow DEBUG: Task 884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/1c1efa15-b7dd-41d7-9d4b-7b74ed23700c_calibrate_945868_41.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/1c1efa15-b7dd-41d7-9d4b-7b74ed23700c_calibrate_945868_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:30 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:30 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:30 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:31 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:31 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:31 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:31 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7fbffb5d36eeeb5afe9d5e86d122f33 with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/42f10031-6b3c-4561-89fa-00ff6b136c9f_calibrate_945867_22.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/42f10031-6b3c-4561-89fa-00ff6b136c9f_calibrate_945867_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:32 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91f5c923fdb5bd554ef3119101514f76 with result from task 1019 parsl.dataflow.dflow DEBUG: Task 1019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/45d0711e-4279-4cbd-8644-a529df22e4ff_calibrate_945867_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/45d0711e-4279-4cbd-8644-a529df22e4ff_calibrate_945867_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:32 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:32 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:33 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:33 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:33 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:33 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:34 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:34 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:34 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:34 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:34 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:35 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:35 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5c478502d9b11e0497907d5d5a4d102 with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 5597951ee7f513af6d1a8cae10a50343 parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246401760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 1104 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f8ec9955-1270-4166-bd83-37f2f6cec7e0_calibrate_945867_41.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f8ec9955-1270-4166-bd83-37f2f6cec7e0_calibrate_945867_41.stderr parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/faf4035b-3520-4717-a863-3e67a2e2f90a_characterizeImage_945867_41.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/faf4035b-3520-4717-a863-3e67a2e2f90a_characterizeImage_945867_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:35 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:35 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:36 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:36 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:36 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:37 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9114dace4b67ef82345c0b47403a72d with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 79eef875071287f112873ad5e657d9b2 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246390680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 1105 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/5b442b66-5ddb-461e-8ac6-3f291ffef0a7_calibrate_945869_54.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/5b442b66-5ddb-461e-8ac6-3f291ffef0a7_calibrate_945869_54.stderr parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/133ce4d9-f333-4bb5-ba7e-e0b15a8bb2d2_characterizeImage_945869_54.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/133ce4d9-f333-4bb5-ba7e-e0b15a8bb2d2_characterizeImage_945869_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:37 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:37 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:38 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:38 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:38 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c21dd59e3017abb9deb240a32129364 with result from task 1064 parsl.dataflow.dflow DEBUG: Task 1064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/ef06335b-9d3a-4659-b18d-e3a87f34bc8b_calibrate_945869_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/ef06335b-9d3a-4659-b18d-e3a87f34bc8b_calibrate_945869_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:38 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:39 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:39 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fafdfc5415121206f313571fb0bbae8c with result from task 794 parsl.dataflow.dflow DEBUG: Task 794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79e05558-647a-4616-9ef6-08b20e566b16_calibrate_945868_21.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79e05558-647a-4616-9ef6-08b20e566b16_calibrate_945868_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:39 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:40 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:40 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:40 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b843b5d8d9b261775f1fb7b3a8d3203b with result from task 913 parsl.dataflow.dflow DEBUG: Task 913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 914 has memoization hash 00c522aea7f210151819d51208c6d4ee parsl.dataflow.memoization INFO: Task 914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 1106 parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1f94cc30-8948-4811-a969-37d0c4b0bba2_calibrate_945869_18.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1f94cc30-8948-4811-a969-37d0c4b0bba2_calibrate_945869_18.stderr parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/dd434d45-d0b9-4fa2-9312-6fdc14033cdf_characterizeImage_945869_18.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/dd434d45-d0b9-4fa2-9312-6fdc14033cdf_characterizeImage_945869_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:40 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:41 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c38473964773c13448dcb173e7298154 with result from task 782 parsl.dataflow.dflow DEBUG: Task 782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/5bdb7516-c129-4555-a41e-998efdcab733_calibrate_945867_44.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/5bdb7516-c129-4555-a41e-998efdcab733_calibrate_945867_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:41 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 39 parsl.dataflow.strategy DEBUG: Executor multi has 39 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 39 parsl.dataflow.strategy DEBUG: Executor multi has 39 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.dflow INFO: Task 1052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7ea3ab2c65f9b87b5ccb002f6314229 with result from task 1052 parsl.dataflow.dflow DEBUG: Task 1052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/55844478-55b4-4394-ad3f-cf3743128ffa_calibrate_945868_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/55844478-55b4-4394-ad3f-cf3743128ffa_calibrate_945868_55.stderr parsl.dataflow.dflow INFO: Task 961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60f0617e669ac4ccefd68e17f9720f52 with result from task 961 parsl.dataflow.dflow DEBUG: Task 961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 962 has memoization hash 5490da9781c35a9682b21c84aa33e0a3 parsl.dataflow.memoization INFO: Task 962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 1107 parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/bc373098-c0da-48bc-ba34-fa85e1223a5a_calibrate_945867_24.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/bc373098-c0da-48bc-ba34-fa85e1223a5a_calibrate_945867_24.stderr parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/74e72023-1221-4174-bcd3-976947e3c1c1_characterizeImage_945867_24.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/74e72023-1221-4174-bcd3-976947e3c1c1_characterizeImage_945867_24.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e088afbd992601a56b294d47c4ce630 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash dd7c3d2f753468968c4053fec5a3f902 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492479963e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 1108 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79c3bb7b-511f-4dd1-8b74-a043523a4e49_calibrate_945868_61.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79c3bb7b-511f-4dd1-8b74-a043523a4e49_calibrate_945868_61.stderr parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/80655bc2-b7f4-4a4a-aeec-0ffdbb0e292d_characterizeImage_945868_61.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/80655bc2-b7f4-4a4a-aeec-0ffdbb0e292d_characterizeImage_945868_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 38 parsl.dataflow.strategy DEBUG: Executor multi has 38 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da2293d29e4d540603424f4b2ab6d852 with result from task 812 parsl.dataflow.dflow DEBUG: Task 812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8dcfb45a-8dc8-4688-b73c-e520bfd6d3e5_calibrate_945868_14.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/8dcfb45a-8dc8-4688-b73c-e520bfd6d3e5_calibrate_945868_14.stderr parsl.dataflow.dflow INFO: Task 983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a005f3c313060d01d5f3965ba1654ff with result from task 983 parsl.dataflow.dflow DEBUG: Task 983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/281c5728-24f4-415b-8aa8-0b1b32c9b525_calibrate_945869_38.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/281c5728-24f4-415b-8aa8-0b1b32c9b525_calibrate_945869_38.stderr parsl.dataflow.dflow INFO: Task 938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 072bb733492875b7c4c44142952073ba with result from task 938 parsl.dataflow.dflow DEBUG: Task 938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/07aada8f-cb8d-45b4-a71b-6f27952d2add_calibrate_945869_43.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/07aada8f-cb8d-45b4-a71b-6f27952d2add_calibrate_945869_43.stderr 2024-03-13 18:39:51 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:39:51 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:52 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:52 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:52 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:52 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:53 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9547075dd1d3de1ec4dc8c533510593 with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/8d59e95d-e0f4-4043-927e-48380c56a18e_calibrate_945757_33.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/8d59e95d-e0f4-4043-927e-48380c56a18e_calibrate_945757_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:54 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:54 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f240812681d6893ce7e38882e126c13 with result from task 929 parsl.dataflow.dflow DEBUG: Task 929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/bbe12b80-0328-47e2-bc8a-304f037e6608_calibrate_945757_44.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945757/bbe12b80-0328-47e2-bc8a-304f037e6608_calibrate_945757_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:54 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:54 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:55 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:55 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:55 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:55 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:56 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:56 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:56 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:56 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 1040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9c257e53b0789e8223377aa1817f7b8 with result from task 1040 parsl.dataflow.dflow DEBUG: Task 1040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/6f1c9aee-33f4-4773-aaf8-7cbf316794a9_calibrate_945867_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/6f1c9aee-33f4-4773-aaf8-7cbf316794a9_calibrate_945867_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:57 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:57 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:57 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:57 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:58 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:58 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:58 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b830b58f81ed859cf01ba5c52a9971ca with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/804ae62a-9ac4-4518-8c14-6e23142a23cf_calibrate_945869_36.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/804ae62a-9ac4-4518-8c14-6e23142a23cf_calibrate_945869_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:58 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:59 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:59 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:59 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:39:59 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d63118bed40363ab6499eae880fa3c3 with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/67e79a2e-ea12-4715-975b-189caf86b1a6_calibrate_945867_43.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/67e79a2e-ea12-4715-975b-189caf86b1a6_calibrate_945867_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:00 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:00 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:00 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:00 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:01 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 1043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13eea29505fdd328b547eeac054c3b64 with result from task 1043 parsl.dataflow.dflow DEBUG: Task 1043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2fe8df07-a9bd-477d-b8ea-4bef5f20582c_calibrate_945868_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/2fe8df07-a9bd-477d-b8ea-4bef5f20582c_calibrate_945868_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:01 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:01 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85302bfe79066072c5dc2251dc154cca with result from task 841 parsl.dataflow.dflow DEBUG: Task 841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 842 has memoization hash 8b24df344966205c47a120d457eb9327 parsl.dataflow.memoization INFO: Task 842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461a8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 1109 parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f444545f-458c-45cb-b1cf-b04e2d7bd3d3_calibrate_945867_54.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f444545f-458c-45cb-b1cf-b04e2d7bd3d3_calibrate_945867_54.stderr parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/567a953f-783e-437c-9a54-696bb38bf863_characterizeImage_945867_54.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/567a953f-783e-437c-9a54-696bb38bf863_characterizeImage_945867_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:01 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:02 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:02 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:02 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c4a665b7debfb58265e475cf61de76d with result from task 967 parsl.dataflow.dflow DEBUG: Task 967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 968 has memoization hash 2e317c9feb0a81903ddd94290e41b9d1 parsl.dataflow.memoization INFO: Task 968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 1110 parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d292a061-354f-4550-b5bb-d1835125407e_calibrate_945867_6.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d292a061-354f-4550-b5bb-d1835125407e_calibrate_945867_6.stderr parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/5a242c61-4811-4646-82cb-0b774cc0bac0_characterizeImage_945867_6.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/5a242c61-4811-4646-82cb-0b774cc0bac0_characterizeImage_945867_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:02 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:03 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:03 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:03 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:03 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:04 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:04 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:04 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:04 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 408e11f901f1d8fc6a40f71e268e61bd with result from task 926 parsl.dataflow.dflow DEBUG: Task 926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/07f39342-4cca-40b5-8efd-ab0d677303a1_calibrate_945868_28.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/07f39342-4cca-40b5-8efd-ab0d677303a1_calibrate_945868_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:05 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cd8fad411e89151602c67a6241ebd9d with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/a1570326-e40c-4661-a5bd-708cc8357d53_calibrate_945867_32.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/a1570326-e40c-4661-a5bd-708cc8357d53_calibrate_945867_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:05 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:05 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:05 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53f0b170682269a7a8336ea2d7d5d994 with result from task 973 parsl.dataflow.dflow DEBUG: Task 973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 974 has memoization hash 4c82a69d36943d450adbb402861edf0d parsl.dataflow.memoization INFO: Task 974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 1111 parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/acc3e3bf-69dd-4780-a66b-564978931765_calibrate_945869_14.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/acc3e3bf-69dd-4780-a66b-564978931765_calibrate_945869_14.stderr parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/fafae265-a6dc-4be3-9b51-110ea2e42065_characterizeImage_945869_14.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/fafae265-a6dc-4be3-9b51-110ea2e42065_characterizeImage_945869_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:06 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bed557de8ff9dd372e04beda15fbb647 with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1effc39e-f9e2-4f24-be3d-d58004c22278_calibrate_945869_35.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1effc39e-f9e2-4f24-be3d-d58004c22278_calibrate_945869_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:06 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:06 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:06 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:07 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:07 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:07 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:07 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:08 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:08 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:08 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3724f3d6ed9f7ff87ea78fabfa18d233 with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/392f6c4a-a9c8-46c1-bd49-c4f1a9ec5f69_calibrate_945869_50.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/392f6c4a-a9c8-46c1-bd49-c4f1a9ec5f69_calibrate_945869_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:08 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:09 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:09 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:09 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:09 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:10 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:10 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:10 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:10 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38bcaa744d81997bb36dc5f0a3a4d858 with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 0c8ae0e53b91adf09fa4c14a5ee41597 parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246402980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 1112 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/7e90b969-73cd-41ef-ab6a-cb63f06565f5_calibrate_945868_44.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/7e90b969-73cd-41ef-ab6a-cb63f06565f5_calibrate_945868_44.stderr parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0c1cc7d0-9ad4-4ee2-aef1-292e0ce53797_characterizeImage_945868_44.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/0c1cc7d0-9ad4-4ee2-aef1-292e0ce53797_characterizeImage_945868_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:11 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:11 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:11 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:11 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:12 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:12 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:12 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:12 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:12 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:13 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:13 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:13 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:13 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0b41ab5e9acf1572ccb4dd4e8c334fd with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/4e3666d4-b01a-4d41-a065-ade98f63d940_calibrate_945868_33.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/4e3666d4-b01a-4d41-a065-ade98f63d940_calibrate_945868_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:14 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:14 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:14 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:14 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.memoization DEBUG: Storing app cache entry 40bcf869020f97601c373f01afc7573d with result from task 917 parsl.dataflow.dflow DEBUG: Task 917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/32004859-a484-4622-9f42-1175684a24ce_calibrate_945867_23.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/32004859-a484-4622-9f42-1175684a24ce_calibrate_945867_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:15 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 23 parsl.dataflow.strategy DEBUG: Executor multi has 23 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 23 parsl.dataflow.strategy DEBUG: Executor multi has 23 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 304cc564e66d4c08aa0902d3121f143a with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash edb77193f8af880140f0e793f13d989c parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492463451c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 1113 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/1d007374-f044-4e39-8326-058538a0e335_calibrate_945755_58.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/1d007374-f044-4e39-8326-058538a0e335_calibrate_945755_58.stderr parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/7def5224-18a5-45a0-9a81-4a3ef64ef58b_characterizeImage_945755_58.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945755/7def5224-18a5-45a0-9a81-4a3ef64ef58b_characterizeImage_945755_58.stderr parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abd4499b9375b213e1b4edb9b0aae3ac with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0f595126-fb35-44ad-92f4-bd00ebc681c8_calibrate_945869_49.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0f595126-fb35-44ad-92f4-bd00ebc681c8_calibrate_945869_49.stderr parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 007dcf03393a78f3b3257c6835121679 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ea21a4b5-a1b4-4914-bf9c-bf63c99f8ad5_calibrate_945868_1.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ea21a4b5-a1b4-4914-bf9c-bf63c99f8ad5_calibrate_945868_1.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 21 parsl.dataflow.strategy DEBUG: Executor multi has 21 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:40:25 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:40:25 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:25 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:25 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:26 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:26 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:26 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:26 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:27 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:27 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:27 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:27 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:28 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:28 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:28 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:28 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:29 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:29 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:29 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:29 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:30 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:30 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:30 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:30 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:31 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:31 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 986ab2d98d480de5c27c3b7627d79696 with result from task 946 parsl.dataflow.dflow DEBUG: Task 946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 947 has memoization hash 2cabc5252db1273780bb2c4d0b0688b3 parsl.dataflow.memoization INFO: Task 947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246229bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 1114 parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/148961f7-a766-4229-b1bc-8693491d1bab_calibrate_945756_32.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/148961f7-a766-4229-b1bc-8693491d1bab_calibrate_945756_32.stderr parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b67267ac-306c-4b76-aea9-f4130a9badb7_characterizeImage_945756_32.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/b67267ac-306c-4b76-aea9-f4130a9badb7_characterizeImage_945756_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:31 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:31 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5597951ee7f513af6d1a8cae10a50343 with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f8ec9955-1270-4166-bd83-37f2f6cec7e0_calibrate_945867_41.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f8ec9955-1270-4166-bd83-37f2f6cec7e0_calibrate_945867_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:32 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:32 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:32 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:33 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:33 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:33 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:33 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79eef875071287f112873ad5e657d9b2 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/5b442b66-5ddb-461e-8ac6-3f291ffef0a7_calibrate_945869_54.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/5b442b66-5ddb-461e-8ac6-3f291ffef0a7_calibrate_945869_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:34 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:34 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:34 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:34 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:34 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:34 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:35 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:35 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:35 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:35 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:36 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:36 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:36 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:36 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:37 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:37 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:37 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 1105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ac2395c0d24c0218e7c7cffd6f68bb2 with result from task 1105 parsl.dataflow.dflow DEBUG: Task 1105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1106 has memoization hash 60a36f2192bdff01fad2d652363ba9f6 parsl.dataflow.memoization INFO: Task 1106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460be160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1106 try 0 launched on executor multi with executor id 1115 parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/726d2d85-9801-44ee-834e-2f403958a203_calibrate_945867_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/726d2d85-9801-44ee-834e-2f403958a203_calibrate_945867_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e1710e07-e59a-4ae1-9e8a-76422176ee15_characterizeImage_945867_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/e1710e07-e59a-4ae1-9e8a-76422176ee15_characterizeImage_945867_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:37 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:38 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7255e7163e603699c90a32cc120b3c56 with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/a3bc32e3-91b4-456a-8623-43a8936b0e08_calibrate_945867_16.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/a3bc32e3-91b4-456a-8623-43a8936b0e08_calibrate_945867_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:38 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:38 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:38 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:39 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:39 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52616753b5069b8b3feab235c22722cf with result from task 977 parsl.dataflow.dflow DEBUG: Task 977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/cb833780-3b41-4d24-b6fa-8d63c277c642_calibrate_945869_17.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/cb833780-3b41-4d24-b6fa-8d63c277c642_calibrate_945869_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:39 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:39 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:40 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:40 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:40 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:40 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:41 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:41 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:41 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:41 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:42 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:42 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:42 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:42 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:43 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:43 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:43 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:43 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:44 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:44 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:44 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:44 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:45 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edb77193f8af880140f0e793f13d989c with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/1d007374-f044-4e39-8326-058538a0e335_calibrate_945755_58.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945755/1d007374-f044-4e39-8326-058538a0e335_calibrate_945755_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:45 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:45 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:45 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:46 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:46 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:46 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:46 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:47 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:47 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:47 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:47 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 16 parsl.dataflow.strategy DEBUG: Executor multi has 16 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 16 parsl.dataflow.strategy DEBUG: Executor multi has 16 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00c522aea7f210151819d51208c6d4ee with result from task 914 parsl.dataflow.dflow DEBUG: Task 914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1f94cc30-8948-4811-a969-37d0c4b0bba2_calibrate_945869_18.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/1f94cc30-8948-4811-a969-37d0c4b0bba2_calibrate_945869_18.stderr parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03ae870062f2e6af0a03a291874580cd with result from task 892 parsl.dataflow.dflow DEBUG: Task 892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 893 has memoization hash eb62df402b6d9c358b1b8db0c9cb2fea parsl.dataflow.memoization INFO: Task 893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492461f0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 1116 parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b4e3c2ce-afae-4849-a042-daf2e3455ecf_calibrate_945868_16.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b4e3c2ce-afae-4849-a042-daf2e3455ecf_calibrate_945868_16.stderr parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a0d58a48-eacf-456b-960d-ba2c91e274b2_characterizeImage_945868_16.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/a0d58a48-eacf-456b-960d-ba2c91e274b2_characterizeImage_945868_16.stderr parsl.dataflow.dflow INFO: Task 962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5490da9781c35a9682b21c84aa33e0a3 with result from task 962 parsl.dataflow.dflow DEBUG: Task 962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/bc373098-c0da-48bc-ba34-fa85e1223a5a_calibrate_945867_24.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/bc373098-c0da-48bc-ba34-fa85e1223a5a_calibrate_945867_24.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 14 parsl.dataflow.strategy DEBUG: Executor multi has 14 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19516bf21694e95df5a619b434167531 with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/d2b0b567-6e54-43bd-afc5-a2579ea4fad2_calibrate_945869_16.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/d2b0b567-6e54-43bd-afc5-a2579ea4fad2_calibrate_945869_16.stderr parsl.dataflow.dflow INFO: Task 1075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdd4b85da71d3ddfaf32b5368639ea8d with result from task 1075 parsl.dataflow.dflow DEBUG: Task 1075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash 7a6035b43b4843871e852f335da07e1c parsl.dataflow.memoization INFO: Task 1076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1492460bc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1076 try 0 launched on executor multi with executor id 1117 parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0ec2b96c-78b2-4f63-97fc-9a96e003794a_calibrate_945869_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0ec2b96c-78b2-4f63-97fc-9a96e003794a_calibrate_945869_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/0e65992e-3e38-41ce-80ad-11c9e6324f4e_characterizeImage_945869_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/0e65992e-3e38-41ce-80ad-11c9e6324f4e_characterizeImage_945869_61.stderr 2024-03-13 18:40:58 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:40:58 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:58 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd7c3d2f753468968c4053fec5a3f902 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79c3bb7b-511f-4dd1-8b74-a043523a4e49_calibrate_945868_61.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/79c3bb7b-511f-4dd1-8b74-a043523a4e49_calibrate_945868_61.stderr parsl.dataflow.dflow INFO: Task 974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c82a69d36943d450adbb402861edf0d with result from task 974 parsl.dataflow.dflow DEBUG: Task 974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/acc3e3bf-69dd-4780-a66b-564978931765_calibrate_945869_14.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/acc3e3bf-69dd-4780-a66b-564978931765_calibrate_945869_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:58 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:59 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:59 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:59 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:40:59 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:00 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:00 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:00 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:00 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:01 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:01 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:01 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:01 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:01 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:02 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:02 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:02 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:02 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:03 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:03 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:03 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:03 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:04 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:04 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:04 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:04 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:05 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:05 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:05 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:05 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:06 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:06 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:06 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:06 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c8ae0e53b91adf09fa4c14a5ee41597 with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/7e90b969-73cd-41ef-ab6a-cb63f06565f5_calibrate_945868_44.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/7e90b969-73cd-41ef-ab6a-cb63f06565f5_calibrate_945868_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:07 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:07 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:07 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:07 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:08 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:08 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cabc5252db1273780bb2c4d0b0688b3 with result from task 947 parsl.dataflow.dflow DEBUG: Task 947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/148961f7-a766-4229-b1bc-8693491d1bab_calibrate_945756_32.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/148961f7-a766-4229-b1bc-8693491d1bab_calibrate_945756_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:08 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:08 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:09 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:09 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:09 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:09 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:10 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:10 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:10 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:10 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:11 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:11 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:11 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:11 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b24df344966205c47a120d457eb9327 with result from task 842 parsl.dataflow.dflow DEBUG: Task 842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f444545f-458c-45cb-b1cf-b04e2d7bd3d3_calibrate_945867_54.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/f444545f-458c-45cb-b1cf-b04e2d7bd3d3_calibrate_945867_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:12 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:12 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:12 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:12 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:13 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:13 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:13 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:13 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:14 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:14 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:14 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:14 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:15 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:15 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:15 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:15 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:16 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:16 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:16 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:16 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:17 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:17 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:17 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:18 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:18 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e317c9feb0a81903ddd94290e41b9d1 with result from task 968 parsl.dataflow.dflow DEBUG: Task 968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d292a061-354f-4550-b5bb-d1835125407e_calibrate_945867_6.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/d292a061-354f-4550-b5bb-d1835125407e_calibrate_945867_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:18 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:18 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:18 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:19 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:19 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:19 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:19 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:20 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:20 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:20 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:20 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:41:30 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:41:31 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:31 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:31 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:31 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:32 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:32 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:32 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:32 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:33 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:33 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:33 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:33 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:34 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:34 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:34 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:34 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:34 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:35 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:35 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:35 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:35 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:36 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:36 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:36 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:36 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:36 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:37 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:37 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:37 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:38 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:38 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:38 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:38 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:38 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:39 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:39 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:39 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:39 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:40 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:40 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:40 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:40 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:41 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:41 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:41 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:41 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:42 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:42 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:42 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:42 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:43 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:43 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:43 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:43 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:44 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:44 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:44 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:44 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:45 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:45 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:45 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:45 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:46 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:46 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:46 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:46 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:47 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:47 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:47 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:47 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:48 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 1106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60a36f2192bdff01fad2d652363ba9f6 with result from task 1106 parsl.dataflow.dflow DEBUG: Task 1106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/726d2d85-9801-44ee-834e-2f403958a203_calibrate_945867_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/726d2d85-9801-44ee-834e-2f403958a203_calibrate_945867_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:48 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:48 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:48 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb62df402b6d9c358b1b8db0c9cb2fea with result from task 893 parsl.dataflow.dflow DEBUG: Task 893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b4e3c2ce-afae-4849-a042-daf2e3455ecf_calibrate_945868_16.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/b4e3c2ce-afae-4849-a042-daf2e3455ecf_calibrate_945868_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:49 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:49 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:49 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:49 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:50 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:50 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:50 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:50 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:51 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:51 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:51 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:51 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:52 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:52 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:52 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:52 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:53 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:41:53 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:42:03 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:42:03 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:04 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:04 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:04 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:04 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:05 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:05 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:05 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:05 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:06 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:06 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:06 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:06 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:06 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:07 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:07 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:07 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:07 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:08 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:08 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:08 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:08 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:09 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:09 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:09 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:09 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:10 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:10 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:10 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:10 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:11 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:11 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:11 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:11 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:12 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:12 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:12 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:12 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:13 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:13 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:13 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:13 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:14 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:14 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:14 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:14 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:15 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:15 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:15 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:15 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:16 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:16 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:16 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:16 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:17 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:17 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:17 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:17 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:18 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:18 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:18 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:18 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:19 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:19 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:19 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:19 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:20 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:20 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:20 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:20 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:21 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:21 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:21 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:21 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:22 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:22 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:22 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:22 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:23 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:23 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:23 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:23 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:24 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:24 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:24 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:24 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:25 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:25 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:25 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:25 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:27 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:27 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27ab555cecedab07cd914481fd6a7296 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 7e6f58491cea5dfdb59fc24c3103ee0d parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924796c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 1118 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/257d3d37-1063-46d1-8a97-21d610619d3f_calibrate_945869_25.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/257d3d37-1063-46d1-8a97-21d610619d3f_calibrate_945869_25.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/b75a3343-fdaf-46ea-8ad4-53cd58148dc2_characterizeImage_945869_25.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945869/b75a3343-fdaf-46ea-8ad4-53cd58148dc2_characterizeImage_945869_25.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:42:37 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:42:37 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:38 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:38 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:38 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:38 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:39 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:39 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:39 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:39 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:40 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:40 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:40 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 1076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a6035b43b4843871e852f335da07e1c with result from task 1076 parsl.dataflow.dflow DEBUG: Task 1076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0ec2b96c-78b2-4f63-97fc-9a96e003794a_calibrate_945869_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/0ec2b96c-78b2-4f63-97fc-9a96e003794a_calibrate_945869_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:40 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:41 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:41 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:41 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:41 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:42 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:42 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:42 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:42 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:43 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:43 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:43 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:43 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:44 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:44 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:44 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:44 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:45 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:45 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:45 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:45 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:46 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:46 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:46 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:46 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:47 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:47 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:47 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:47 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:48 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:48 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:48 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:48 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:48 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:49 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:49 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:49 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:49 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:50 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:50 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:50 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:50 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:51 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:51 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:51 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:51 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:51 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:52 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:52 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:52 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:52 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:53 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:53 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:53 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:53 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:54 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:54 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:54 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:54 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:55 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:55 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:55 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:55 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:56 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:56 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:56 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:56 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:57 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:58 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:58 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:59 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:59 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:59 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:42:59 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:00 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:00 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:00 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:00 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:01 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:01 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:43:11 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:43:11 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:11 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:12 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:12 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:12 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:12 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:13 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:13 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:13 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:13 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:14 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:14 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:14 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:14 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:15 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:15 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:15 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:15 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:16 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:16 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:16 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:16 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:17 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:17 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:17 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:17 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:18 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:18 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:18 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:18 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:19 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:19 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:19 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:19 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e6f58491cea5dfdb59fc24c3103ee0d with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/257d3d37-1063-46d1-8a97-21d610619d3f_calibrate_945869_25.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945869/257d3d37-1063-46d1-8a97-21d610619d3f_calibrate_945869_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:20 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:20 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:20 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:20 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:21 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:21 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:21 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:21 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:22 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:22 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:22 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:22 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:23 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:23 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:23 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:23 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:24 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:24 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:24 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cd49ee39b329dd62f8f26a8276dedc2 with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash a11d8b7bc1c6944105de300c20a757e7 parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246403a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 1119 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/46a4fac7-f8be-41b1-b07d-40248da2e3ff_calibrate_945867_25.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/46a4fac7-f8be-41b1-b07d-40248da2e3ff_calibrate_945867_25.stderr parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/b018b229-3e5d-48f2-b019-278224ccb377_characterizeImage_945867_25.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945867/b018b229-3e5d-48f2-b019-278224ccb377_characterizeImage_945867_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:24 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:25 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:25 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:25 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:25 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:25 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:26 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:26 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:26 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:26 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:27 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:27 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:27 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:27 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:28 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:28 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:28 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:28 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:29 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:29 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:29 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:29 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:30 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:30 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:30 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:30 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:31 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:31 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:31 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:31 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:32 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:32 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:32 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:32 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:33 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:33 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:33 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:33 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:34 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:43:44 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:43:44 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:44 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:44 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:45 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:45 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:45 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:45 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:46 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:46 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:46 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:46 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:47 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:47 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:47 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:47 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:48 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:48 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:48 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:48 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:49 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:49 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:49 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:49 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:50 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:50 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:50 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:50 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:51 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:51 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:51 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:51 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:52 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:52 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:52 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:52 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:53 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:53 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:53 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:53 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:54 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:54 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:54 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:54 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:55 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:55 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:55 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:55 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:56 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:56 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:56 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:56 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:57 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:57 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:57 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:57 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:58 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:58 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:58 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:58 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:59 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:59 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:59 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:43:59 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:00 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:00 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:00 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:00 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:01 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:01 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:01 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:01 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:02 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:02 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:02 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:02 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:03 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:03 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:03 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:03 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:04 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:04 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:04 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:04 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:05 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:05 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:05 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:05 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:06 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:06 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:06 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:06 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:07 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:44:17 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:44:17 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:17 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:17 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:18 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:18 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:18 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a11d8b7bc1c6944105de300c20a757e7 with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/46a4fac7-f8be-41b1-b07d-40248da2e3ff_calibrate_945867_25.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945867/46a4fac7-f8be-41b1-b07d-40248da2e3ff_calibrate_945867_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:18 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:19 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:19 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:19 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:19 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:20 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:20 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:20 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:20 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:20 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:21 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:21 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:21 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:21 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:22 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:22 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:22 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:22 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:23 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:23 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:23 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:23 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:24 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:24 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:24 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:24 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:25 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:25 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:25 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:25 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:26 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:26 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:26 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:26 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:27 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:27 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:27 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:27 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:28 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:28 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:28 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:28 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:29 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:29 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:29 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:29 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:30 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:30 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:30 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:30 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:31 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:31 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:31 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:31 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:32 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:32 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:32 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:32 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:33 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:33 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:33 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:33 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:34 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:34 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:34 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:34 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:34 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:35 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:35 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:35 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:35 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:36 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:36 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:36 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:36 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:37 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:37 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:37 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:37 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:38 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:38 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:38 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:38 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:39 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:39 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:39 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbe49ba270a93f612fe24544eadc07b8 with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash 0d23186f364b9a6b1f0708a49bf8f9f0 parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149246403ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 1120 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ea30919c-8239-4591-a068-53595bdf6c1b_calibrate_945868_25.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ea30919c-8239-4591-a068-53595bdf6c1b_calibrate_945868_25.stderr parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/046cffaf-4595-4f4a-a83e-5b6b79d83d51_characterizeImage_945868_25.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945868/046cffaf-4595-4f4a-a83e-5b6b79d83d51_characterizeImage_945868_25.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:44:49 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:44:49 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:50 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:50 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:50 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:50 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:51 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:51 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:51 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:51 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:52 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:52 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:52 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:52 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:53 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:53 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:53 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:53 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:54 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:54 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:54 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:54 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:55 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:55 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:55 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:55 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:56 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:56 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:56 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:56 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:57 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:57 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:57 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:57 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:58 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:58 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:58 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:59 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:59 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:59 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:44:59 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:00 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:00 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:00 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:00 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:01 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:01 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:01 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:01 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:02 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:02 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:02 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:02 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:03 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:03 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:03 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:03 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:04 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:04 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:04 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:04 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:05 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:05 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:05 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:05 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:06 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:06 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:06 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:06 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:07 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:07 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:07 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:07 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:08 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:08 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:08 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:08 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:09 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:09 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:09 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:09 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:10 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:10 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:10 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:10 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:11 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:11 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:11 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:11 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:11 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:12 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:12 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:12 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:45:22 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:45:23 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:23 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:23 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:23 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:23 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:24 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:24 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:24 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:24 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:25 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:25 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:25 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:25 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:26 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:26 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:26 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:26 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:27 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:27 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:27 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:27 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:28 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:28 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:28 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:28 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:29 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:29 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:29 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:29 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:30 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:30 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:30 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:30 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:31 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:31 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:31 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:31 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:32 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:32 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:32 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:32 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:33 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:33 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:33 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:33 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:34 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:34 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:34 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:34 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:34 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:35 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:35 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:35 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:35 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:36 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:36 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:36 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:36 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:37 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:37 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:37 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:37 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:38 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:38 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:38 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:38 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:39 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:39 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:39 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:39 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:40 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:40 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:40 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:40 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:41 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:41 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:41 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:41 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:42 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:42 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:42 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:42 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:43 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:43 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:43 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:43 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:44 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:44 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:44 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:44 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:45 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:45 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d23186f364b9a6b1f0708a49bf8f9f0 with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ea30919c-8239-4591-a068-53595bdf6c1b_calibrate_945868_25.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945868/ea30919c-8239-4591-a068-53595bdf6c1b_calibrate_945868_25.stderr 2024-03-13 18:45:55 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:45:55 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:55 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:56 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:56 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:56 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:56 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:57 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:57 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:57 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:57 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:58 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:58 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:58 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:58 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:59 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:59 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:59 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:45:59 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:00 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:00 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:00 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:00 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:01 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:01 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:01 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:01 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:02 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:02 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:02 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:02 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:03 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:03 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:03 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:03 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:04 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:04 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:04 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:04 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:05 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:05 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:05 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:05 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:06 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:06 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:06 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:06 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:06 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:07 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:07 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:07 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:07 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:08 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:08 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:08 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:08 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:09 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:09 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:09 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:09 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:10 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:10 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:10 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:10 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:11 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:11 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:11 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:11 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:12 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:12 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:12 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:12 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:13 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:13 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:13 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:13 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:14 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:14 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:14 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:14 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:15 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:15 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:15 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:15 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:16 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:16 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:16 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:16 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:17 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:17 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:17 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:17 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:18 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:46:28 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:46:28 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:28 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:28 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:29 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:30 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:30 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:30 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:30 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:31 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:31 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:31 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:31 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:32 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:32 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:32 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:32 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:33 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:33 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:33 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:33 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:34 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:34 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:34 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:34 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:34 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:35 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:35 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:35 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:35 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:36 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:36 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:36 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:36 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8bd9c3a01f10e208a79bf18c6fdf29b with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash ce6f0720d2817c5247c2f7961507ad08 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14924792db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 1121 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/40dbe8bd-ba50-4509-8aae-3a80bd490782_calibrate_945756_11.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/40dbe8bd-ba50-4509-8aae-3a80bd490782_calibrate_945756_11.stderr parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/cc5bd271-be5a-4243-bd4c-7d3cddfb58c2_characterizeImage_945756_11.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/characterizeImage/945756/cc5bd271-be5a-4243-bd4c-7d3cddfb58c2_characterizeImage_945756_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:37 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:37 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:37 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:37 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:37 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:38 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:38 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:38 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:38 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:38 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:39 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:39 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:39 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:39 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:39 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:39 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:40 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:40 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:40 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:40 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:40 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:41 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:41 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:41 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:41 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:41 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:42 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:42 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:42 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:42 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:42 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:43 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:43 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:43 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:43 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:43 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:44 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:44 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:44 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:44 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:44 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:45 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:45 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:45 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:45 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:45 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:46 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:46 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:46 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:46 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:46 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:47 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:47 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:47 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:47 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:47 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:48 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:48 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:48 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 2024-03-13 18:46:58 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:46:58 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:58 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:59 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:59 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:59 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:59 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:46:59 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:00 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:01 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:01 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:01 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:01 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:01 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:02 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:02 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:02 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:02 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:02 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:03 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:03 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:03 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:03 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:03 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:04 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:04 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:04 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:04 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:04 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:05 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:05 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020626,17020627,17020628,17020629,17020630,17020631,17020632,17020633,17020634,17020635,17020636,17020637,17020638,17020639,17020640,17020641,17020642,17020643,17020644,17020645,17020646,17020647,17020648,17020649,17020650,17020651,17020652,17020653,17020654,17020655' parsl.providers.slurm.slurm DEBUG: squeue returned 17020652 R 17020653 R 17020654 R 17020655 R 17020642 R 17020643 R 17020644 R 17020645 R 17020646 R 17020647 R 17020648 R 17020649 R 17020650 R 17020651 R 17020635 R 17020636 R 17020637 R 17020638 R 17020639 R 17020640 R 17020641 R 17020628 R 17020629 R 17020630 R 17020631 R 17020632 R 17020633 R 17020634 R 17020626 R 17020627 R parsl.providers.slurm.slurm DEBUG: Updating job 17020652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020627 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:05 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:05 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:05 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:06 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:06 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020656,17020657,17020658,17020659,17020660' parsl.providers.slurm.slurm DEBUG: squeue returned 17020660 R 17020656 R 17020657 R 17020658 R 17020659 R parsl.providers.slurm.slurm DEBUG: Updating job 17020660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020659 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:06 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:06 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:06 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:07 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:07 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:07 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:07 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:07 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:08 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:08 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:08 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:08 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:08 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:09 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:09 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:09 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:09 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:09 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:10 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:10 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020661,17020662,17020663,17020664,17020665,17020666,17020667,17020668,17020669,17020670,17020671,17020672,17020673,17020674,17020675,17020676,17020678,17020679,17020680,17020681' parsl.providers.slurm.slurm DEBUG: squeue returned 17020680 R 17020681 R 17020673 R 17020674 R 17020675 R 17020676 R 17020678 R 17020679 R 17020665 R 17020666 R 17020667 R 17020668 R 17020669 R 17020670 R 17020671 R 17020672 R 17020661 R 17020662 R 17020663 R 17020664 R parsl.providers.slurm.slurm DEBUG: Updating job 17020680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020679 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020664 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:10 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:10 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:10 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:10 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:11 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:11 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:11 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:11 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:11 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:12 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:12 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:12 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:12 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:12 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:13 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:13 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:13 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:13 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:13 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:14 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:14 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:14 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:14 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:14 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:15 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:15 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:15 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:15 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:15 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:16 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:16 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:16 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:16 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:16 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:17 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:17 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING 2024-03-13 18:47:17 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17020682,17020683,17020684,17020685,17020686,17020687,17020688,17020689,17020690,17020691,17020692,17020693,17020694,17020695,17020696,17020697,17020698,17020699,17020700,17020701,17020702,17020703,17020704,17020705,17020706,17020707,17020708,17020709,17020710,17020711,17020712,17020713,17020714,17020715,17020716,17020717,17020718' parsl.providers.slurm.slurm DEBUG: squeue returned 17020716 R 17020717 R 17020718 R 17020695 R 17020696 R 17020697 R 17020698 R 17020699 R 17020700 R 17020701 R 17020702 R 17020703 R 17020704 R 17020705 R 17020706 R 17020707 R 17020708 R 17020709 R 17020710 R 17020711 R 17020712 R 17020713 R 17020714 R 17020715 R 17020682 R 17020683 R 17020684 R 17020685 R 17020686 R 17020687 R 17020688 R 17020689 R 17020690 R 17020691 R 17020692 R 17020693 R 17020694 R parsl.providers.slurm.slurm DEBUG: Updating job 17020716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020682 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17020694 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22618516612304 parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce6f0720d2817c5247c2f7961507ad08 with result from task 62 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 1093 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.failed: 14 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 9 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/40dbe8bd-ba50-4509-8aae-3a80bd490782_calibrate_945756_11.stdout parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/submit/DEEP/20201015/B1e/science#step1/20240313T183244Z/logs/calibrate/945756/40dbe8bd-ba50-4509-8aae-3a80bd490782_calibrate_945756_11.stderr parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 93 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 74340ca0e7f5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 74340ca0e7f5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c036c76f2182 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c036c76f2182 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e63a515c6986 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e63a515c6986 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fac57466d3a5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fac57466d3a5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 69e05cf97e2b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 69e05cf97e2b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 393f9b194c16 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 393f9b194c16 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d25440a0fbbd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d25440a0fbbd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fff625838b25 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fff625838b25 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c77a6f55fe75 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c77a6f55fe75 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5cfbf5475461 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5cfbf5475461 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 458673e58316 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 458673e58316 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 15fad081de9f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 15fad081de9f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 240751dcfaed parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 240751dcfaed parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 19af13bbaa14 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 19af13bbaa14 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff846966daaf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff846966daaf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f1831765025c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f1831765025c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e3f901b1af3e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e3f901b1af3e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: beab057d4502 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: beab057d4502 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 44fe4b5ca711 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 44fe4b5ca711 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5fec6a18d2be parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5fec6a18d2be parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4e70a2e55299 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4e70a2e55299 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 14febfabc236 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 14febfabc236 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5ea5c68ab28c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5ea5c68ab28c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a9aff45aaa16 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a9aff45aaa16 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fed8267fe856 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fed8267fe856 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dc267faf2932 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dc267faf2932 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 20d43f595c52 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 20d43f595c52 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0a1942f34303 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0a1942f34303 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 337293a4dbcb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 337293a4dbcb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d47c8fb70802 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d47c8fb70802 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 44140d0425e9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 44140d0425e9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bc110f9b3beb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bc110f9b3beb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 35543f4fa5df parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 35543f4fa5df parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e557abc6e0e9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e557abc6e0e9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7ce578f81911 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7ce578f81911 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cb6b9bda0423 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cb6b9bda0423 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dcb5e8fc0d77 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dcb5e8fc0d77 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 451760bf0ef9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 451760bf0ef9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e80aee3c5ec1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e80aee3c5ec1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 76543f83dfd9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 76543f83dfd9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 58f1672e222d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 58f1672e222d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1ed2c98fb47b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1ed2c98fb47b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a9a85f617160 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a9a85f617160 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5a69af667b38 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5a69af667b38 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 534ec91fc9dd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 534ec91fc9dd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 692ee985344d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 692ee985344d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0f741107d84e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0f741107d84e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2f46c55d12d0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2f46c55d12d0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 24e6b9b90e5f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 24e6b9b90e5f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b2af21e9a98d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b2af21e9a98d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9d2adaf9342b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9d2adaf9342b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 701f05dca20c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 701f05dca20c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8ffe4f3fb3a2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8ffe4f3fb3a2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c166412c7738 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c166412c7738 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ace6a1518b5e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ace6a1518b5e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d6d3f88bccf8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d6d3f88bccf8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d17ffa71390 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d17ffa71390 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6095267914c7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6095267914c7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 69408844f592 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 69408844f592 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c9f3c77f56fe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c9f3c77f56fe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 97e969c23a42 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 97e969c23a42 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a84a97d06ff7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a84a97d06ff7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0fac9ba835e9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0fac9ba835e9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a6304e56aa53 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a6304e56aa53 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d4cc140eeb64 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d4cc140eeb64 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 31026ce68cf2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 31026ce68cf2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 45363116a716 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 45363116a716 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1ae18bb59c10 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1ae18bb59c10 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ef844c146f74 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ef844c146f74 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d026bb93e55 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d026bb93e55 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 93412fc1222b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 93412fc1222b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7b1f587ea1ea parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7b1f587ea1ea parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f9d639ce4828 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f9d639ce4828 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 82ccafc37ab2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 82ccafc37ab2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 109ddc872b1f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 109ddc872b1f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4f5a32fb98a7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4f5a32fb98a7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 832c2b29f825 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 832c2b29f825 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: daee6d983586 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: daee6d983586 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1dd2ea9ebb7e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1dd2ea9ebb7e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 10d8926d6270 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 10d8926d6270 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dc731339760c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dc731339760c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8ad7217bc568 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8ad7217bc568 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d2526c4dec52 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d2526c4dec52 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: eddc8c535bd7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: eddc8c535bd7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5fd3ee397cd2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5fd3ee397cd2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 78b93b6f3e23 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 78b93b6f3e23 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bfd47d30d82e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bfd47d30d82e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2529a61172aa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2529a61172aa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c9f6724e77b4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c9f6724e77b4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1b69c928d298 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1b69c928d298 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6f94d5db5b18 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6f94d5db5b18 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 14424b127003 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 14424b127003 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f773260686a2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f773260686a2 2024-03-13 18:47:21 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint 2024-03-13 18:47:21 proc_lsst.multi:201 [INFO] cancelling 17020714 on provider checkpoint proc_lsst.multi INFO: cancelling 17020714 on provider checkpoint 2024-03-13 18:47:21 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro 2024-03-13 18:47:21 proc_lsst.multi:201 [INFO] cancelling 17020627 on provider astro proc_lsst.multi INFO: cancelling 17020627 on provider astro 2024-03-13 18:47:21 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint 2024-03-13 18:47:21 proc_lsst.multi:201 [INFO] cancelling 17020704 on provider checkpoint proc_lsst.multi INFO: cancelling 17020704 on provider checkpoint 2024-03-13 18:47:22 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint 2024-03-13 18:47:22 proc_lsst.multi:201 [INFO] cancelling 17020685 on provider checkpoint proc_lsst.multi INFO: cancelling 17020685 on provider checkpoint 2024-03-13 18:47:22 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint 2024-03-13 18:47:22 proc_lsst.multi:201 [INFO] cancelling 17020694 on provider checkpoint proc_lsst.multi INFO: cancelling 17020694 on provider checkpoint 2024-03-13 18:47:22 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro 2024-03-13 18:47:22 proc_lsst.multi:201 [INFO] cancelling 17020655 on provider astro proc_lsst.multi INFO: cancelling 17020655 on provider astro 2024-03-13 18:47:22 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro 2024-03-13 18:47:22 proc_lsst.multi:201 [INFO] cancelling 17020643 on provider astro proc_lsst.multi INFO: cancelling 17020643 on provider astro 2024-03-13 18:47:22 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 2024-03-13 18:47:22 proc_lsst.multi:201 [INFO] cancelling 17020676 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020676 on provider escience_a40 2024-03-13 18:47:22 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro 2024-03-13 18:47:22 proc_lsst.multi:201 [INFO] cancelling 17020646 on provider astro proc_lsst.multi INFO: cancelling 17020646 on provider astro 2024-03-13 18:47:23 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro 2024-03-13 18:47:23 proc_lsst.multi:201 [INFO] cancelling 17020647 on provider astro proc_lsst.multi INFO: cancelling 17020647 on provider astro 2024-03-13 18:47:23 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 2024-03-13 18:47:23 proc_lsst.multi:201 [INFO] cancelling 17020674 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020674 on provider escience_a40 2024-03-13 18:47:23 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 2024-03-13 18:47:23 proc_lsst.multi:201 [INFO] cancelling 17020678 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020678 on provider escience_a40 2024-03-13 18:47:23 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro 2024-03-13 18:47:23 proc_lsst.multi:201 [INFO] cancelling 17020650 on provider astro proc_lsst.multi INFO: cancelling 17020650 on provider astro 2024-03-13 18:47:23 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro 2024-03-13 18:47:23 proc_lsst.multi:201 [INFO] cancelling 17020634 on provider astro proc_lsst.multi INFO: cancelling 17020634 on provider astro 2024-03-13 18:47:23 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint 2024-03-13 18:47:23 proc_lsst.multi:201 [INFO] cancelling 17020717 on provider checkpoint proc_lsst.multi INFO: cancelling 17020717 on provider checkpoint 2024-03-13 18:47:24 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint 2024-03-13 18:47:24 proc_lsst.multi:201 [INFO] cancelling 17020689 on provider checkpoint proc_lsst.multi INFO: cancelling 17020689 on provider checkpoint 2024-03-13 18:47:24 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint 2024-03-13 18:47:24 proc_lsst.multi:201 [INFO] cancelling 17020690 on provider checkpoint proc_lsst.multi INFO: cancelling 17020690 on provider checkpoint 2024-03-13 18:47:24 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint 2024-03-13 18:47:24 proc_lsst.multi:201 [INFO] cancelling 17020709 on provider checkpoint proc_lsst.multi INFO: cancelling 17020709 on provider checkpoint 2024-03-13 18:47:24 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint 2024-03-13 18:47:24 proc_lsst.multi:201 [INFO] cancelling 17020713 on provider checkpoint proc_lsst.multi INFO: cancelling 17020713 on provider checkpoint 2024-03-13 18:47:24 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k 2024-03-13 18:47:24 proc_lsst.multi:201 [INFO] cancelling 17020658 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17020658 on provider escience_rtx6k 2024-03-13 18:47:24 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint 2024-03-13 18:47:24 proc_lsst.multi:201 [INFO] cancelling 17020705 on provider checkpoint proc_lsst.multi INFO: cancelling 17020705 on provider checkpoint 2024-03-13 18:47:25 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint 2024-03-13 18:47:25 proc_lsst.multi:201 [INFO] cancelling 17020696 on provider checkpoint proc_lsst.multi INFO: cancelling 17020696 on provider checkpoint 2024-03-13 18:47:25 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro 2024-03-13 18:47:25 proc_lsst.multi:201 [INFO] cancelling 17020638 on provider astro proc_lsst.multi INFO: cancelling 17020638 on provider astro 2024-03-13 18:47:25 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro 2024-03-13 18:47:25 proc_lsst.multi:201 [INFO] cancelling 17020653 on provider astro proc_lsst.multi INFO: cancelling 17020653 on provider astro 2024-03-13 18:47:25 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint 2024-03-13 18:47:25 proc_lsst.multi:201 [INFO] cancelling 17020700 on provider checkpoint proc_lsst.multi INFO: cancelling 17020700 on provider checkpoint 2024-03-13 18:47:25 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 2024-03-13 18:47:25 proc_lsst.multi:201 [INFO] cancelling 17020675 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020675 on provider escience_a40 2024-03-13 18:47:25 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint 2024-03-13 18:47:25 proc_lsst.multi:201 [INFO] cancelling 17020703 on provider checkpoint proc_lsst.multi INFO: cancelling 17020703 on provider checkpoint 2024-03-13 18:47:25 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint 2024-03-13 18:47:25 proc_lsst.multi:201 [INFO] cancelling 17020686 on provider checkpoint proc_lsst.multi INFO: cancelling 17020686 on provider checkpoint 2024-03-13 18:47:26 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint 2024-03-13 18:47:26 proc_lsst.multi:201 [INFO] cancelling 17020706 on provider checkpoint proc_lsst.multi INFO: cancelling 17020706 on provider checkpoint 2024-03-13 18:47:26 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint 2024-03-13 18:47:26 proc_lsst.multi:201 [INFO] cancelling 17020692 on provider checkpoint proc_lsst.multi INFO: cancelling 17020692 on provider checkpoint 2024-03-13 18:47:26 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro 2024-03-13 18:47:26 proc_lsst.multi:201 [INFO] cancelling 17020648 on provider astro proc_lsst.multi INFO: cancelling 17020648 on provider astro 2024-03-13 18:47:26 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint 2024-03-13 18:47:26 proc_lsst.multi:201 [INFO] cancelling 17020702 on provider checkpoint proc_lsst.multi INFO: cancelling 17020702 on provider checkpoint 2024-03-13 18:47:26 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint 2024-03-13 18:47:26 proc_lsst.multi:201 [INFO] cancelling 17020710 on provider checkpoint proc_lsst.multi INFO: cancelling 17020710 on provider checkpoint 2024-03-13 18:47:26 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro 2024-03-13 18:47:26 proc_lsst.multi:201 [INFO] cancelling 17020636 on provider astro proc_lsst.multi INFO: cancelling 17020636 on provider astro 2024-03-13 18:47:27 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 2024-03-13 18:47:27 proc_lsst.multi:201 [INFO] cancelling 17020672 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020672 on provider escience_a40 2024-03-13 18:47:27 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint 2024-03-13 18:47:27 proc_lsst.multi:201 [INFO] cancelling 17020682 on provider checkpoint proc_lsst.multi INFO: cancelling 17020682 on provider checkpoint 2024-03-13 18:47:27 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 2024-03-13 18:47:27 proc_lsst.multi:201 [INFO] cancelling 17020667 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020667 on provider escience_a40 2024-03-13 18:47:27 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro 2024-03-13 18:47:27 proc_lsst.multi:201 [INFO] cancelling 17020626 on provider astro proc_lsst.multi INFO: cancelling 17020626 on provider astro 2024-03-13 18:47:27 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint 2024-03-13 18:47:27 proc_lsst.multi:201 [INFO] cancelling 17020707 on provider checkpoint proc_lsst.multi INFO: cancelling 17020707 on provider checkpoint 2024-03-13 18:47:27 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint 2024-03-13 18:47:27 proc_lsst.multi:201 [INFO] cancelling 17020697 on provider checkpoint proc_lsst.multi INFO: cancelling 17020697 on provider checkpoint 2024-03-13 18:47:28 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro 2024-03-13 18:47:28 proc_lsst.multi:201 [INFO] cancelling 17020651 on provider astro proc_lsst.multi INFO: cancelling 17020651 on provider astro 2024-03-13 18:47:28 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint 2024-03-13 18:47:28 proc_lsst.multi:201 [INFO] cancelling 17020701 on provider checkpoint proc_lsst.multi INFO: cancelling 17020701 on provider checkpoint 2024-03-13 18:47:28 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint 2024-03-13 18:47:28 proc_lsst.multi:201 [INFO] cancelling 17020691 on provider checkpoint proc_lsst.multi INFO: cancelling 17020691 on provider checkpoint 2024-03-13 18:47:28 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint 2024-03-13 18:47:28 proc_lsst.multi:201 [INFO] cancelling 17020715 on provider checkpoint proc_lsst.multi INFO: cancelling 17020715 on provider checkpoint 2024-03-13 18:47:28 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint 2024-03-13 18:47:28 proc_lsst.multi:201 [INFO] cancelling 17020687 on provider checkpoint proc_lsst.multi INFO: cancelling 17020687 on provider checkpoint 2024-03-13 18:47:28 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 2024-03-13 18:47:28 proc_lsst.multi:201 [INFO] cancelling 17020665 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020665 on provider escience_a40 2024-03-13 18:47:29 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 2024-03-13 18:47:29 proc_lsst.multi:201 [INFO] cancelling 17020679 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020679 on provider escience_a40 2024-03-13 18:47:29 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 2024-03-13 18:47:29 proc_lsst.multi:201 [INFO] cancelling 17020673 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020673 on provider escience_a40 2024-03-13 18:47:29 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint 2024-03-13 18:47:29 proc_lsst.multi:201 [INFO] cancelling 17020716 on provider checkpoint proc_lsst.multi INFO: cancelling 17020716 on provider checkpoint 2024-03-13 18:47:29 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro 2024-03-13 18:47:29 proc_lsst.multi:201 [INFO] cancelling 17020640 on provider astro proc_lsst.multi INFO: cancelling 17020640 on provider astro 2024-03-13 18:47:29 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro 2024-03-13 18:47:29 proc_lsst.multi:201 [INFO] cancelling 17020641 on provider astro proc_lsst.multi INFO: cancelling 17020641 on provider astro 2024-03-13 18:47:29 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint 2024-03-13 18:47:29 proc_lsst.multi:201 [INFO] cancelling 17020684 on provider checkpoint proc_lsst.multi INFO: cancelling 17020684 on provider checkpoint 2024-03-13 18:47:30 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint 2024-03-13 18:47:30 proc_lsst.multi:201 [INFO] cancelling 17020683 on provider checkpoint proc_lsst.multi INFO: cancelling 17020683 on provider checkpoint 2024-03-13 18:47:30 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k 2024-03-13 18:47:30 proc_lsst.multi:201 [INFO] cancelling 17020656 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17020656 on provider escience_rtx6k 2024-03-13 18:47:30 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint 2024-03-13 18:47:30 proc_lsst.multi:201 [INFO] cancelling 17020695 on provider checkpoint proc_lsst.multi INFO: cancelling 17020695 on provider checkpoint 2024-03-13 18:47:30 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k 2024-03-13 18:47:30 proc_lsst.multi:201 [INFO] cancelling 17020659 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17020659 on provider escience_rtx6k 2024-03-13 18:47:30 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint 2024-03-13 18:47:30 proc_lsst.multi:201 [INFO] cancelling 17020693 on provider checkpoint proc_lsst.multi INFO: cancelling 17020693 on provider checkpoint 2024-03-13 18:47:31 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint 2024-03-13 18:47:31 proc_lsst.multi:201 [INFO] cancelling 17020711 on provider checkpoint proc_lsst.multi INFO: cancelling 17020711 on provider checkpoint 2024-03-13 18:47:31 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro 2024-03-13 18:47:31 proc_lsst.multi:201 [INFO] cancelling 17020649 on provider astro proc_lsst.multi INFO: cancelling 17020649 on provider astro 2024-03-13 18:47:32 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint 2024-03-13 18:47:32 proc_lsst.multi:201 [INFO] cancelling 17020718 on provider checkpoint proc_lsst.multi INFO: cancelling 17020718 on provider checkpoint 2024-03-13 18:47:32 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 2024-03-13 18:47:32 proc_lsst.multi:201 [INFO] cancelling 17020662 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020662 on provider escience_a40 2024-03-13 18:47:32 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint 2024-03-13 18:47:32 proc_lsst.multi:201 [INFO] cancelling 17020699 on provider checkpoint proc_lsst.multi INFO: cancelling 17020699 on provider checkpoint 2024-03-13 18:47:32 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 2024-03-13 18:47:32 proc_lsst.multi:201 [INFO] cancelling 17020666 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020666 on provider escience_a40 2024-03-13 18:47:32 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro 2024-03-13 18:47:32 proc_lsst.multi:201 [INFO] cancelling 17020633 on provider astro proc_lsst.multi INFO: cancelling 17020633 on provider astro 2024-03-13 18:47:32 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro 2024-03-13 18:47:32 proc_lsst.multi:201 [INFO] cancelling 17020642 on provider astro proc_lsst.multi INFO: cancelling 17020642 on provider astro 2024-03-13 18:47:32 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 2024-03-13 18:47:32 proc_lsst.multi:201 [INFO] cancelling 17020670 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020670 on provider escience_a40 2024-03-13 18:47:33 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro 2024-03-13 18:47:33 proc_lsst.multi:201 [INFO] cancelling 17020628 on provider astro proc_lsst.multi INFO: cancelling 17020628 on provider astro 2024-03-13 18:47:33 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro 2024-03-13 18:47:33 proc_lsst.multi:201 [INFO] cancelling 17020654 on provider astro proc_lsst.multi INFO: cancelling 17020654 on provider astro 2024-03-13 18:47:33 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro 2024-03-13 18:47:33 proc_lsst.multi:201 [INFO] cancelling 17020644 on provider astro proc_lsst.multi INFO: cancelling 17020644 on provider astro 2024-03-13 18:47:33 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro 2024-03-13 18:47:33 proc_lsst.multi:201 [INFO] cancelling 17020629 on provider astro proc_lsst.multi INFO: cancelling 17020629 on provider astro 2024-03-13 18:47:33 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro 2024-03-13 18:47:33 proc_lsst.multi:201 [INFO] cancelling 17020631 on provider astro proc_lsst.multi INFO: cancelling 17020631 on provider astro 2024-03-13 18:47:34 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro 2024-03-13 18:47:34 proc_lsst.multi:201 [INFO] cancelling 17020637 on provider astro proc_lsst.multi INFO: cancelling 17020637 on provider astro 2024-03-13 18:47:34 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:47:34 proc_lsst.multi:201 [INFO] cancelling 98521 on provider local proc_lsst.multi INFO: cancelling 98521 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 98521 2024-03-13 18:47:34 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 2024-03-13 18:47:34 proc_lsst.multi:201 [INFO] cancelling 17020661 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020661 on provider escience_a40 2024-03-13 18:47:34 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 2024-03-13 18:47:34 proc_lsst.multi:201 [INFO] cancelling 17020669 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020669 on provider escience_a40 2024-03-13 18:47:34 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 2024-03-13 18:47:34 proc_lsst.multi:201 [INFO] cancelling 17020663 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020663 on provider escience_a40 2024-03-13 18:47:34 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro 2024-03-13 18:47:34 proc_lsst.multi:201 [INFO] cancelling 17020639 on provider astro proc_lsst.multi INFO: cancelling 17020639 on provider astro 2024-03-13 18:47:34 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k 2024-03-13 18:47:34 proc_lsst.multi:201 [INFO] cancelling 17020660 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17020660 on provider escience_rtx6k 2024-03-13 18:47:35 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint 2024-03-13 18:47:35 proc_lsst.multi:201 [INFO] cancelling 17020698 on provider checkpoint proc_lsst.multi INFO: cancelling 17020698 on provider checkpoint 2024-03-13 18:47:35 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro 2024-03-13 18:47:35 proc_lsst.multi:201 [INFO] cancelling 17020630 on provider astro proc_lsst.multi INFO: cancelling 17020630 on provider astro 2024-03-13 18:47:35 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 2024-03-13 18:47:35 proc_lsst.multi:201 [INFO] cancelling 17020671 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020671 on provider escience_a40 2024-03-13 18:47:35 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro 2024-03-13 18:47:35 proc_lsst.multi:201 [INFO] cancelling 17020632 on provider astro proc_lsst.multi INFO: cancelling 17020632 on provider astro 2024-03-13 18:47:35 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint 2024-03-13 18:47:35 proc_lsst.multi:201 [INFO] cancelling 17020708 on provider checkpoint proc_lsst.multi INFO: cancelling 17020708 on provider checkpoint 2024-03-13 18:47:35 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint 2024-03-13 18:47:35 proc_lsst.multi:201 [INFO] cancelling 17020688 on provider checkpoint proc_lsst.multi INFO: cancelling 17020688 on provider checkpoint 2024-03-13 18:47:35 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro 2024-03-13 18:47:35 proc_lsst.multi:201 [INFO] cancelling 17020652 on provider astro proc_lsst.multi INFO: cancelling 17020652 on provider astro 2024-03-13 18:47:36 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint 2024-03-13 18:47:36 proc_lsst.multi:201 [INFO] cancelling 17020712 on provider checkpoint proc_lsst.multi INFO: cancelling 17020712 on provider checkpoint 2024-03-13 18:47:36 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k 2024-03-13 18:47:36 proc_lsst.multi:201 [INFO] cancelling 17020657 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17020657 on provider escience_rtx6k 2024-03-13 18:47:36 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 2024-03-13 18:47:36 proc_lsst.multi:201 [INFO] cancelling 17020681 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020681 on provider escience_a40 2024-03-13 18:47:36 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 2024-03-13 18:47:36 proc_lsst.multi:201 [INFO] cancelling 17020664 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020664 on provider escience_a40 2024-03-13 18:47:36 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 2024-03-13 18:47:36 proc_lsst.multi:201 [INFO] cancelling 17020680 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020680 on provider escience_a40 2024-03-13 18:47:36 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro 2024-03-13 18:47:36 proc_lsst.multi:201 [INFO] cancelling 17020635 on provider astro proc_lsst.multi INFO: cancelling 17020635 on provider astro 2024-03-13 18:47:37 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 2024-03-13 18:47:37 proc_lsst.multi:201 [INFO] cancelling 17020668 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020668 on provider escience_a40 2024-03-13 18:47:37 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro 2024-03-13 18:47:37 proc_lsst.multi:201 [INFO] cancelling 17020645 on provider astro proc_lsst.multi INFO: cancelling 17020645 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-13 18:47:37 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-13 18:47:37 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17020682', '17020683', '17020684', '17020685', '17020686', '17020687', '17020688', '17020689', '17020690', '17020691', '17020692', '17020693', '17020694', '17020695', '17020696', '17020697', '17020698', '17020699', '17020700', '17020701', '17020702', '17020703', '17020704', '17020705', '17020706', '17020707', '17020708', '17020709', '17020710', '17020711', '17020712', '17020713', '17020714', '17020715', '17020716', '17020717', '17020718', '17020626', '17020627', '17020628', '17020629', '17020630', '17020631', '17020632', '17020633', '17020634', '17020635', '17020636', '17020637', '17020638', '17020639', '17020640', '17020641', '17020642', '17020643', '17020644', '17020645', '17020646', '17020647', '17020648', '17020649', '17020650', '17020651', '17020652', '17020653', '17020654', '17020655', '17020661', '17020662', '17020663', '17020664', '17020665', '17020666', '17020667', '17020668', '17020669', '17020670', '17020671', '17020672', '17020673', '17020674', '17020675', '17020676', '17020678', '17020679', '17020680', '17020681', '17020656', '17020657', '17020658', '17020659', '17020660', '98521'] proc_lsst.multi INFO: new jobs since last cancel ['17020682', '17020683', '17020684', '17020685', '17020686', '17020687', '17020688', '17020689', '17020690', '17020691', '17020692', '17020693', '17020694', '17020695', '17020696', '17020697', '17020698', '17020699', '17020700', '17020701', '17020702', '17020703', '17020704', '17020705', '17020706', '17020707', '17020708', '17020709', '17020710', '17020711', '17020712', '17020713', '17020714', '17020715', '17020716', '17020717', '17020718', '17020626', '17020627', '17020628', '17020629', '17020630', '17020631', '17020632', '17020633', '17020634', '17020635', '17020636', '17020637', '17020638', '17020639', '17020640', '17020641', '17020642', '17020643', '17020644', '17020645', '17020646', '17020647', '17020648', '17020649', '17020650', '17020651', '17020652', '17020653', '17020654', '17020655', '17020661', '17020662', '17020663', '17020664', '17020665', '17020666', '17020667', '17020668', '17020669', '17020670', '17020671', '17020672', '17020673', '17020674', '17020675', '17020676', '17020678', '17020679', '17020680', '17020681', '17020656', '17020657', '17020658', '17020659', '17020660', '98521'] 2024-03-13 18:47:37 proc_lsst.multi:146 [INFO] found job 17020682 in provider checkpoint proc_lsst.multi INFO: found job 17020682 in provider checkpoint 2024-03-13 18:47:37 proc_lsst.multi:201 [INFO] cancelling 17020682 on provider checkpoint proc_lsst.multi INFO: cancelling 17020682 on provider checkpoint 2024-03-13 18:47:37 proc_lsst.multi:146 [INFO] found job 17020683 in provider checkpoint proc_lsst.multi INFO: found job 17020683 in provider checkpoint 2024-03-13 18:47:37 proc_lsst.multi:201 [INFO] cancelling 17020683 on provider checkpoint proc_lsst.multi INFO: cancelling 17020683 on provider checkpoint 2024-03-13 18:47:37 proc_lsst.multi:146 [INFO] found job 17020684 in provider checkpoint proc_lsst.multi INFO: found job 17020684 in provider checkpoint 2024-03-13 18:47:37 proc_lsst.multi:201 [INFO] cancelling 17020684 on provider checkpoint proc_lsst.multi INFO: cancelling 17020684 on provider checkpoint 2024-03-13 18:47:38 proc_lsst.multi:146 [INFO] found job 17020685 in provider checkpoint proc_lsst.multi INFO: found job 17020685 in provider checkpoint 2024-03-13 18:47:38 proc_lsst.multi:201 [INFO] cancelling 17020685 on provider checkpoint proc_lsst.multi INFO: cancelling 17020685 on provider checkpoint 2024-03-13 18:47:38 proc_lsst.multi:146 [INFO] found job 17020686 in provider checkpoint proc_lsst.multi INFO: found job 17020686 in provider checkpoint 2024-03-13 18:47:38 proc_lsst.multi:201 [INFO] cancelling 17020686 on provider checkpoint proc_lsst.multi INFO: cancelling 17020686 on provider checkpoint 2024-03-13 18:47:38 proc_lsst.multi:146 [INFO] found job 17020687 in provider checkpoint proc_lsst.multi INFO: found job 17020687 in provider checkpoint 2024-03-13 18:47:38 proc_lsst.multi:201 [INFO] cancelling 17020687 on provider checkpoint proc_lsst.multi INFO: cancelling 17020687 on provider checkpoint 2024-03-13 18:47:38 proc_lsst.multi:146 [INFO] found job 17020688 in provider checkpoint proc_lsst.multi INFO: found job 17020688 in provider checkpoint 2024-03-13 18:47:38 proc_lsst.multi:201 [INFO] cancelling 17020688 on provider checkpoint proc_lsst.multi INFO: cancelling 17020688 on provider checkpoint 2024-03-13 18:47:38 proc_lsst.multi:146 [INFO] found job 17020689 in provider checkpoint proc_lsst.multi INFO: found job 17020689 in provider checkpoint 2024-03-13 18:47:38 proc_lsst.multi:201 [INFO] cancelling 17020689 on provider checkpoint proc_lsst.multi INFO: cancelling 17020689 on provider checkpoint 2024-03-13 18:47:38 proc_lsst.multi:146 [INFO] found job 17020690 in provider checkpoint proc_lsst.multi INFO: found job 17020690 in provider checkpoint 2024-03-13 18:47:38 proc_lsst.multi:201 [INFO] cancelling 17020690 on provider checkpoint proc_lsst.multi INFO: cancelling 17020690 on provider checkpoint 2024-03-13 18:47:39 proc_lsst.multi:146 [INFO] found job 17020691 in provider checkpoint proc_lsst.multi INFO: found job 17020691 in provider checkpoint 2024-03-13 18:47:39 proc_lsst.multi:201 [INFO] cancelling 17020691 on provider checkpoint proc_lsst.multi INFO: cancelling 17020691 on provider checkpoint 2024-03-13 18:47:39 proc_lsst.multi:146 [INFO] found job 17020692 in provider checkpoint proc_lsst.multi INFO: found job 17020692 in provider checkpoint 2024-03-13 18:47:39 proc_lsst.multi:201 [INFO] cancelling 17020692 on provider checkpoint proc_lsst.multi INFO: cancelling 17020692 on provider checkpoint 2024-03-13 18:47:39 proc_lsst.multi:146 [INFO] found job 17020693 in provider checkpoint proc_lsst.multi INFO: found job 17020693 in provider checkpoint 2024-03-13 18:47:39 proc_lsst.multi:201 [INFO] cancelling 17020693 on provider checkpoint proc_lsst.multi INFO: cancelling 17020693 on provider checkpoint 2024-03-13 18:47:39 proc_lsst.multi:146 [INFO] found job 17020694 in provider checkpoint proc_lsst.multi INFO: found job 17020694 in provider checkpoint 2024-03-13 18:47:39 proc_lsst.multi:201 [INFO] cancelling 17020694 on provider checkpoint proc_lsst.multi INFO: cancelling 17020694 on provider checkpoint 2024-03-13 18:47:39 proc_lsst.multi:146 [INFO] found job 17020695 in provider checkpoint proc_lsst.multi INFO: found job 17020695 in provider checkpoint 2024-03-13 18:47:39 proc_lsst.multi:201 [INFO] cancelling 17020695 on provider checkpoint proc_lsst.multi INFO: cancelling 17020695 on provider checkpoint 2024-03-13 18:47:39 proc_lsst.multi:146 [INFO] found job 17020696 in provider checkpoint proc_lsst.multi INFO: found job 17020696 in provider checkpoint 2024-03-13 18:47:39 proc_lsst.multi:201 [INFO] cancelling 17020696 on provider checkpoint proc_lsst.multi INFO: cancelling 17020696 on provider checkpoint 2024-03-13 18:47:40 proc_lsst.multi:146 [INFO] found job 17020697 in provider checkpoint proc_lsst.multi INFO: found job 17020697 in provider checkpoint 2024-03-13 18:47:40 proc_lsst.multi:201 [INFO] cancelling 17020697 on provider checkpoint proc_lsst.multi INFO: cancelling 17020697 on provider checkpoint 2024-03-13 18:47:40 proc_lsst.multi:146 [INFO] found job 17020698 in provider checkpoint proc_lsst.multi INFO: found job 17020698 in provider checkpoint 2024-03-13 18:47:40 proc_lsst.multi:201 [INFO] cancelling 17020698 on provider checkpoint proc_lsst.multi INFO: cancelling 17020698 on provider checkpoint 2024-03-13 18:47:40 proc_lsst.multi:146 [INFO] found job 17020699 in provider checkpoint proc_lsst.multi INFO: found job 17020699 in provider checkpoint 2024-03-13 18:47:40 proc_lsst.multi:201 [INFO] cancelling 17020699 on provider checkpoint proc_lsst.multi INFO: cancelling 17020699 on provider checkpoint 2024-03-13 18:47:40 proc_lsst.multi:146 [INFO] found job 17020700 in provider checkpoint proc_lsst.multi INFO: found job 17020700 in provider checkpoint 2024-03-13 18:47:40 proc_lsst.multi:201 [INFO] cancelling 17020700 on provider checkpoint proc_lsst.multi INFO: cancelling 17020700 on provider checkpoint 2024-03-13 18:47:40 proc_lsst.multi:146 [INFO] found job 17020701 in provider checkpoint proc_lsst.multi INFO: found job 17020701 in provider checkpoint 2024-03-13 18:47:40 proc_lsst.multi:201 [INFO] cancelling 17020701 on provider checkpoint proc_lsst.multi INFO: cancelling 17020701 on provider checkpoint 2024-03-13 18:47:40 proc_lsst.multi:146 [INFO] found job 17020702 in provider checkpoint proc_lsst.multi INFO: found job 17020702 in provider checkpoint 2024-03-13 18:47:40 proc_lsst.multi:201 [INFO] cancelling 17020702 on provider checkpoint proc_lsst.multi INFO: cancelling 17020702 on provider checkpoint 2024-03-13 18:47:41 proc_lsst.multi:146 [INFO] found job 17020703 in provider checkpoint proc_lsst.multi INFO: found job 17020703 in provider checkpoint 2024-03-13 18:47:41 proc_lsst.multi:201 [INFO] cancelling 17020703 on provider checkpoint proc_lsst.multi INFO: cancelling 17020703 on provider checkpoint 2024-03-13 18:47:41 proc_lsst.multi:146 [INFO] found job 17020704 in provider checkpoint proc_lsst.multi INFO: found job 17020704 in provider checkpoint 2024-03-13 18:47:41 proc_lsst.multi:201 [INFO] cancelling 17020704 on provider checkpoint proc_lsst.multi INFO: cancelling 17020704 on provider checkpoint 2024-03-13 18:47:41 proc_lsst.multi:146 [INFO] found job 17020705 in provider checkpoint proc_lsst.multi INFO: found job 17020705 in provider checkpoint 2024-03-13 18:47:41 proc_lsst.multi:201 [INFO] cancelling 17020705 on provider checkpoint proc_lsst.multi INFO: cancelling 17020705 on provider checkpoint 2024-03-13 18:47:41 proc_lsst.multi:146 [INFO] found job 17020706 in provider checkpoint proc_lsst.multi INFO: found job 17020706 in provider checkpoint 2024-03-13 18:47:41 proc_lsst.multi:201 [INFO] cancelling 17020706 on provider checkpoint proc_lsst.multi INFO: cancelling 17020706 on provider checkpoint 2024-03-13 18:47:41 proc_lsst.multi:146 [INFO] found job 17020707 in provider checkpoint proc_lsst.multi INFO: found job 17020707 in provider checkpoint 2024-03-13 18:47:41 proc_lsst.multi:201 [INFO] cancelling 17020707 on provider checkpoint proc_lsst.multi INFO: cancelling 17020707 on provider checkpoint 2024-03-13 18:47:41 proc_lsst.multi:146 [INFO] found job 17020708 in provider checkpoint proc_lsst.multi INFO: found job 17020708 in provider checkpoint 2024-03-13 18:47:41 proc_lsst.multi:201 [INFO] cancelling 17020708 on provider checkpoint proc_lsst.multi INFO: cancelling 17020708 on provider checkpoint 2024-03-13 18:47:42 proc_lsst.multi:146 [INFO] found job 17020709 in provider checkpoint proc_lsst.multi INFO: found job 17020709 in provider checkpoint 2024-03-13 18:47:42 proc_lsst.multi:201 [INFO] cancelling 17020709 on provider checkpoint proc_lsst.multi INFO: cancelling 17020709 on provider checkpoint 2024-03-13 18:47:42 proc_lsst.multi:146 [INFO] found job 17020710 in provider checkpoint proc_lsst.multi INFO: found job 17020710 in provider checkpoint 2024-03-13 18:47:42 proc_lsst.multi:201 [INFO] cancelling 17020710 on provider checkpoint proc_lsst.multi INFO: cancelling 17020710 on provider checkpoint 2024-03-13 18:47:42 proc_lsst.multi:146 [INFO] found job 17020711 in provider checkpoint proc_lsst.multi INFO: found job 17020711 in provider checkpoint 2024-03-13 18:47:42 proc_lsst.multi:201 [INFO] cancelling 17020711 on provider checkpoint proc_lsst.multi INFO: cancelling 17020711 on provider checkpoint 2024-03-13 18:47:42 proc_lsst.multi:146 [INFO] found job 17020712 in provider checkpoint proc_lsst.multi INFO: found job 17020712 in provider checkpoint 2024-03-13 18:47:42 proc_lsst.multi:201 [INFO] cancelling 17020712 on provider checkpoint proc_lsst.multi INFO: cancelling 17020712 on provider checkpoint 2024-03-13 18:47:42 proc_lsst.multi:146 [INFO] found job 17020713 in provider checkpoint proc_lsst.multi INFO: found job 17020713 in provider checkpoint 2024-03-13 18:47:42 proc_lsst.multi:201 [INFO] cancelling 17020713 on provider checkpoint proc_lsst.multi INFO: cancelling 17020713 on provider checkpoint 2024-03-13 18:47:43 proc_lsst.multi:146 [INFO] found job 17020714 in provider checkpoint proc_lsst.multi INFO: found job 17020714 in provider checkpoint 2024-03-13 18:47:43 proc_lsst.multi:201 [INFO] cancelling 17020714 on provider checkpoint proc_lsst.multi INFO: cancelling 17020714 on provider checkpoint 2024-03-13 18:47:43 proc_lsst.multi:146 [INFO] found job 17020715 in provider checkpoint proc_lsst.multi INFO: found job 17020715 in provider checkpoint 2024-03-13 18:47:43 proc_lsst.multi:201 [INFO] cancelling 17020715 on provider checkpoint proc_lsst.multi INFO: cancelling 17020715 on provider checkpoint 2024-03-13 18:47:43 proc_lsst.multi:146 [INFO] found job 17020716 in provider checkpoint proc_lsst.multi INFO: found job 17020716 in provider checkpoint 2024-03-13 18:47:43 proc_lsst.multi:201 [INFO] cancelling 17020716 on provider checkpoint proc_lsst.multi INFO: cancelling 17020716 on provider checkpoint 2024-03-13 18:47:43 proc_lsst.multi:146 [INFO] found job 17020717 in provider checkpoint proc_lsst.multi INFO: found job 17020717 in provider checkpoint 2024-03-13 18:47:43 proc_lsst.multi:201 [INFO] cancelling 17020717 on provider checkpoint proc_lsst.multi INFO: cancelling 17020717 on provider checkpoint 2024-03-13 18:47:43 proc_lsst.multi:146 [INFO] found job 17020718 in provider checkpoint proc_lsst.multi INFO: found job 17020718 in provider checkpoint 2024-03-13 18:47:43 proc_lsst.multi:201 [INFO] cancelling 17020718 on provider checkpoint proc_lsst.multi INFO: cancelling 17020718 on provider checkpoint 2024-03-13 18:47:43 proc_lsst.multi:146 [INFO] found job 17020626 in provider astro proc_lsst.multi INFO: found job 17020626 in provider astro 2024-03-13 18:47:43 proc_lsst.multi:201 [INFO] cancelling 17020626 on provider astro proc_lsst.multi INFO: cancelling 17020626 on provider astro 2024-03-13 18:47:44 proc_lsst.multi:146 [INFO] found job 17020627 in provider astro proc_lsst.multi INFO: found job 17020627 in provider astro 2024-03-13 18:47:44 proc_lsst.multi:201 [INFO] cancelling 17020627 on provider astro proc_lsst.multi INFO: cancelling 17020627 on provider astro 2024-03-13 18:47:44 proc_lsst.multi:146 [INFO] found job 17020628 in provider astro proc_lsst.multi INFO: found job 17020628 in provider astro 2024-03-13 18:47:44 proc_lsst.multi:201 [INFO] cancelling 17020628 on provider astro proc_lsst.multi INFO: cancelling 17020628 on provider astro 2024-03-13 18:47:44 proc_lsst.multi:146 [INFO] found job 17020629 in provider astro proc_lsst.multi INFO: found job 17020629 in provider astro 2024-03-13 18:47:44 proc_lsst.multi:201 [INFO] cancelling 17020629 on provider astro proc_lsst.multi INFO: cancelling 17020629 on provider astro 2024-03-13 18:47:44 proc_lsst.multi:146 [INFO] found job 17020630 in provider astro proc_lsst.multi INFO: found job 17020630 in provider astro 2024-03-13 18:47:44 proc_lsst.multi:201 [INFO] cancelling 17020630 on provider astro proc_lsst.multi INFO: cancelling 17020630 on provider astro 2024-03-13 18:47:44 proc_lsst.multi:146 [INFO] found job 17020631 in provider astro proc_lsst.multi INFO: found job 17020631 in provider astro 2024-03-13 18:47:44 proc_lsst.multi:201 [INFO] cancelling 17020631 on provider astro proc_lsst.multi INFO: cancelling 17020631 on provider astro 2024-03-13 18:47:44 proc_lsst.multi:146 [INFO] found job 17020632 in provider astro proc_lsst.multi INFO: found job 17020632 in provider astro 2024-03-13 18:47:44 proc_lsst.multi:201 [INFO] cancelling 17020632 on provider astro proc_lsst.multi INFO: cancelling 17020632 on provider astro 2024-03-13 18:47:45 proc_lsst.multi:146 [INFO] found job 17020633 in provider astro proc_lsst.multi INFO: found job 17020633 in provider astro 2024-03-13 18:47:45 proc_lsst.multi:201 [INFO] cancelling 17020633 on provider astro proc_lsst.multi INFO: cancelling 17020633 on provider astro 2024-03-13 18:47:45 proc_lsst.multi:146 [INFO] found job 17020634 in provider astro proc_lsst.multi INFO: found job 17020634 in provider astro 2024-03-13 18:47:45 proc_lsst.multi:201 [INFO] cancelling 17020634 on provider astro proc_lsst.multi INFO: cancelling 17020634 on provider astro 2024-03-13 18:47:45 proc_lsst.multi:146 [INFO] found job 17020635 in provider astro proc_lsst.multi INFO: found job 17020635 in provider astro 2024-03-13 18:47:45 proc_lsst.multi:201 [INFO] cancelling 17020635 on provider astro proc_lsst.multi INFO: cancelling 17020635 on provider astro 2024-03-13 18:47:45 proc_lsst.multi:146 [INFO] found job 17020636 in provider astro proc_lsst.multi INFO: found job 17020636 in provider astro 2024-03-13 18:47:45 proc_lsst.multi:201 [INFO] cancelling 17020636 on provider astro proc_lsst.multi INFO: cancelling 17020636 on provider astro 2024-03-13 18:47:45 proc_lsst.multi:146 [INFO] found job 17020637 in provider astro proc_lsst.multi INFO: found job 17020637 in provider astro 2024-03-13 18:47:45 proc_lsst.multi:201 [INFO] cancelling 17020637 on provider astro proc_lsst.multi INFO: cancelling 17020637 on provider astro 2024-03-13 18:47:45 proc_lsst.multi:146 [INFO] found job 17020638 in provider astro proc_lsst.multi INFO: found job 17020638 in provider astro 2024-03-13 18:47:45 proc_lsst.multi:201 [INFO] cancelling 17020638 on provider astro proc_lsst.multi INFO: cancelling 17020638 on provider astro 2024-03-13 18:47:46 proc_lsst.multi:146 [INFO] found job 17020639 in provider astro proc_lsst.multi INFO: found job 17020639 in provider astro 2024-03-13 18:47:46 proc_lsst.multi:201 [INFO] cancelling 17020639 on provider astro proc_lsst.multi INFO: cancelling 17020639 on provider astro 2024-03-13 18:47:46 proc_lsst.multi:146 [INFO] found job 17020640 in provider astro proc_lsst.multi INFO: found job 17020640 in provider astro 2024-03-13 18:47:46 proc_lsst.multi:201 [INFO] cancelling 17020640 on provider astro proc_lsst.multi INFO: cancelling 17020640 on provider astro 2024-03-13 18:47:46 proc_lsst.multi:146 [INFO] found job 17020641 in provider astro proc_lsst.multi INFO: found job 17020641 in provider astro 2024-03-13 18:47:46 proc_lsst.multi:201 [INFO] cancelling 17020641 on provider astro proc_lsst.multi INFO: cancelling 17020641 on provider astro 2024-03-13 18:47:46 proc_lsst.multi:146 [INFO] found job 17020642 in provider astro proc_lsst.multi INFO: found job 17020642 in provider astro 2024-03-13 18:47:46 proc_lsst.multi:201 [INFO] cancelling 17020642 on provider astro proc_lsst.multi INFO: cancelling 17020642 on provider astro 2024-03-13 18:47:46 proc_lsst.multi:146 [INFO] found job 17020643 in provider astro proc_lsst.multi INFO: found job 17020643 in provider astro 2024-03-13 18:47:46 proc_lsst.multi:201 [INFO] cancelling 17020643 on provider astro proc_lsst.multi INFO: cancelling 17020643 on provider astro 2024-03-13 18:47:46 proc_lsst.multi:146 [INFO] found job 17020644 in provider astro proc_lsst.multi INFO: found job 17020644 in provider astro 2024-03-13 18:47:46 proc_lsst.multi:201 [INFO] cancelling 17020644 on provider astro proc_lsst.multi INFO: cancelling 17020644 on provider astro 2024-03-13 18:47:47 proc_lsst.multi:146 [INFO] found job 17020645 in provider astro proc_lsst.multi INFO: found job 17020645 in provider astro 2024-03-13 18:47:47 proc_lsst.multi:201 [INFO] cancelling 17020645 on provider astro proc_lsst.multi INFO: cancelling 17020645 on provider astro 2024-03-13 18:47:47 proc_lsst.multi:146 [INFO] found job 17020646 in provider astro proc_lsst.multi INFO: found job 17020646 in provider astro 2024-03-13 18:47:47 proc_lsst.multi:201 [INFO] cancelling 17020646 on provider astro proc_lsst.multi INFO: cancelling 17020646 on provider astro 2024-03-13 18:47:47 proc_lsst.multi:146 [INFO] found job 17020647 in provider astro proc_lsst.multi INFO: found job 17020647 in provider astro 2024-03-13 18:47:47 proc_lsst.multi:201 [INFO] cancelling 17020647 on provider astro proc_lsst.multi INFO: cancelling 17020647 on provider astro 2024-03-13 18:47:47 proc_lsst.multi:146 [INFO] found job 17020648 in provider astro proc_lsst.multi INFO: found job 17020648 in provider astro 2024-03-13 18:47:47 proc_lsst.multi:201 [INFO] cancelling 17020648 on provider astro proc_lsst.multi INFO: cancelling 17020648 on provider astro 2024-03-13 18:47:47 proc_lsst.multi:146 [INFO] found job 17020649 in provider astro proc_lsst.multi INFO: found job 17020649 in provider astro 2024-03-13 18:47:47 proc_lsst.multi:201 [INFO] cancelling 17020649 on provider astro proc_lsst.multi INFO: cancelling 17020649 on provider astro 2024-03-13 18:47:47 proc_lsst.multi:146 [INFO] found job 17020650 in provider astro proc_lsst.multi INFO: found job 17020650 in provider astro 2024-03-13 18:47:47 proc_lsst.multi:201 [INFO] cancelling 17020650 on provider astro proc_lsst.multi INFO: cancelling 17020650 on provider astro 2024-03-13 18:47:48 proc_lsst.multi:146 [INFO] found job 17020651 in provider astro proc_lsst.multi INFO: found job 17020651 in provider astro 2024-03-13 18:47:48 proc_lsst.multi:201 [INFO] cancelling 17020651 on provider astro proc_lsst.multi INFO: cancelling 17020651 on provider astro 2024-03-13 18:47:48 proc_lsst.multi:146 [INFO] found job 17020652 in provider astro proc_lsst.multi INFO: found job 17020652 in provider astro 2024-03-13 18:47:48 proc_lsst.multi:201 [INFO] cancelling 17020652 on provider astro proc_lsst.multi INFO: cancelling 17020652 on provider astro 2024-03-13 18:47:48 proc_lsst.multi:146 [INFO] found job 17020653 in provider astro proc_lsst.multi INFO: found job 17020653 in provider astro 2024-03-13 18:47:48 proc_lsst.multi:201 [INFO] cancelling 17020653 on provider astro proc_lsst.multi INFO: cancelling 17020653 on provider astro 2024-03-13 18:47:48 proc_lsst.multi:146 [INFO] found job 17020654 in provider astro proc_lsst.multi INFO: found job 17020654 in provider astro 2024-03-13 18:47:48 proc_lsst.multi:201 [INFO] cancelling 17020654 on provider astro proc_lsst.multi INFO: cancelling 17020654 on provider astro 2024-03-13 18:47:48 proc_lsst.multi:146 [INFO] found job 17020655 in provider astro proc_lsst.multi INFO: found job 17020655 in provider astro 2024-03-13 18:47:48 proc_lsst.multi:201 [INFO] cancelling 17020655 on provider astro proc_lsst.multi INFO: cancelling 17020655 on provider astro 2024-03-13 18:47:49 proc_lsst.multi:146 [INFO] found job 17020661 in provider escience_a40 proc_lsst.multi INFO: found job 17020661 in provider escience_a40 2024-03-13 18:47:49 proc_lsst.multi:201 [INFO] cancelling 17020661 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020661 on provider escience_a40 2024-03-13 18:47:49 proc_lsst.multi:146 [INFO] found job 17020662 in provider escience_a40 proc_lsst.multi INFO: found job 17020662 in provider escience_a40 2024-03-13 18:47:49 proc_lsst.multi:201 [INFO] cancelling 17020662 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020662 on provider escience_a40 2024-03-13 18:47:49 proc_lsst.multi:146 [INFO] found job 17020663 in provider escience_a40 proc_lsst.multi INFO: found job 17020663 in provider escience_a40 2024-03-13 18:47:49 proc_lsst.multi:201 [INFO] cancelling 17020663 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020663 on provider escience_a40 2024-03-13 18:47:49 proc_lsst.multi:146 [INFO] found job 17020664 in provider escience_a40 proc_lsst.multi INFO: found job 17020664 in provider escience_a40 2024-03-13 18:47:49 proc_lsst.multi:201 [INFO] cancelling 17020664 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020664 on provider escience_a40 2024-03-13 18:47:49 proc_lsst.multi:146 [INFO] found job 17020665 in provider escience_a40 proc_lsst.multi INFO: found job 17020665 in provider escience_a40 2024-03-13 18:47:49 proc_lsst.multi:201 [INFO] cancelling 17020665 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020665 on provider escience_a40 2024-03-13 18:47:49 proc_lsst.multi:146 [INFO] found job 17020666 in provider escience_a40 proc_lsst.multi INFO: found job 17020666 in provider escience_a40 2024-03-13 18:47:49 proc_lsst.multi:201 [INFO] cancelling 17020666 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020666 on provider escience_a40 2024-03-13 18:47:50 proc_lsst.multi:146 [INFO] found job 17020667 in provider escience_a40 proc_lsst.multi INFO: found job 17020667 in provider escience_a40 2024-03-13 18:47:50 proc_lsst.multi:201 [INFO] cancelling 17020667 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020667 on provider escience_a40 2024-03-13 18:47:50 proc_lsst.multi:146 [INFO] found job 17020668 in provider escience_a40 proc_lsst.multi INFO: found job 17020668 in provider escience_a40 2024-03-13 18:47:50 proc_lsst.multi:201 [INFO] cancelling 17020668 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020668 on provider escience_a40 2024-03-13 18:47:50 proc_lsst.multi:146 [INFO] found job 17020669 in provider escience_a40 proc_lsst.multi INFO: found job 17020669 in provider escience_a40 2024-03-13 18:47:50 proc_lsst.multi:201 [INFO] cancelling 17020669 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020669 on provider escience_a40 2024-03-13 18:47:50 proc_lsst.multi:146 [INFO] found job 17020670 in provider escience_a40 proc_lsst.multi INFO: found job 17020670 in provider escience_a40 2024-03-13 18:47:50 proc_lsst.multi:201 [INFO] cancelling 17020670 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020670 on provider escience_a40 2024-03-13 18:47:50 proc_lsst.multi:146 [INFO] found job 17020671 in provider escience_a40 proc_lsst.multi INFO: found job 17020671 in provider escience_a40 2024-03-13 18:47:50 proc_lsst.multi:201 [INFO] cancelling 17020671 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020671 on provider escience_a40 2024-03-13 18:47:50 proc_lsst.multi:146 [INFO] found job 17020672 in provider escience_a40 proc_lsst.multi INFO: found job 17020672 in provider escience_a40 2024-03-13 18:47:50 proc_lsst.multi:201 [INFO] cancelling 17020672 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020672 on provider escience_a40 2024-03-13 18:47:51 proc_lsst.multi:146 [INFO] found job 17020673 in provider escience_a40 proc_lsst.multi INFO: found job 17020673 in provider escience_a40 2024-03-13 18:47:51 proc_lsst.multi:201 [INFO] cancelling 17020673 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020673 on provider escience_a40 2024-03-13 18:47:51 proc_lsst.multi:146 [INFO] found job 17020674 in provider escience_a40 proc_lsst.multi INFO: found job 17020674 in provider escience_a40 2024-03-13 18:47:51 proc_lsst.multi:201 [INFO] cancelling 17020674 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020674 on provider escience_a40 2024-03-13 18:47:51 proc_lsst.multi:146 [INFO] found job 17020675 in provider escience_a40 proc_lsst.multi INFO: found job 17020675 in provider escience_a40 2024-03-13 18:47:51 proc_lsst.multi:201 [INFO] cancelling 17020675 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020675 on provider escience_a40 2024-03-13 18:47:51 proc_lsst.multi:146 [INFO] found job 17020676 in provider escience_a40 proc_lsst.multi INFO: found job 17020676 in provider escience_a40 2024-03-13 18:47:51 proc_lsst.multi:201 [INFO] cancelling 17020676 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020676 on provider escience_a40 2024-03-13 18:47:51 proc_lsst.multi:146 [INFO] found job 17020678 in provider escience_a40 proc_lsst.multi INFO: found job 17020678 in provider escience_a40 2024-03-13 18:47:51 proc_lsst.multi:201 [INFO] cancelling 17020678 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020678 on provider escience_a40 2024-03-13 18:47:51 proc_lsst.multi:146 [INFO] found job 17020679 in provider escience_a40 proc_lsst.multi INFO: found job 17020679 in provider escience_a40 2024-03-13 18:47:51 proc_lsst.multi:201 [INFO] cancelling 17020679 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020679 on provider escience_a40 2024-03-13 18:47:52 proc_lsst.multi:146 [INFO] found job 17020680 in provider escience_a40 proc_lsst.multi INFO: found job 17020680 in provider escience_a40 2024-03-13 18:47:52 proc_lsst.multi:201 [INFO] cancelling 17020680 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020680 on provider escience_a40 2024-03-13 18:47:52 proc_lsst.multi:146 [INFO] found job 17020681 in provider escience_a40 proc_lsst.multi INFO: found job 17020681 in provider escience_a40 2024-03-13 18:47:52 proc_lsst.multi:201 [INFO] cancelling 17020681 on provider escience_a40 proc_lsst.multi INFO: cancelling 17020681 on provider escience_a40 2024-03-13 18:47:52 proc_lsst.multi:146 [INFO] found job 17020656 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020656 in provider escience_rtx6k 2024-03-13 18:47:52 proc_lsst.multi:201 [INFO] cancelling 17020656 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17020656 on provider escience_rtx6k 2024-03-13 18:47:52 proc_lsst.multi:146 [INFO] found job 17020657 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020657 in provider escience_rtx6k 2024-03-13 18:47:52 proc_lsst.multi:201 [INFO] cancelling 17020657 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17020657 on provider escience_rtx6k 2024-03-13 18:47:52 proc_lsst.multi:146 [INFO] found job 17020658 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020658 in provider escience_rtx6k 2024-03-13 18:47:52 proc_lsst.multi:201 [INFO] cancelling 17020658 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17020658 on provider escience_rtx6k 2024-03-13 18:47:52 proc_lsst.multi:146 [INFO] found job 17020659 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020659 in provider escience_rtx6k 2024-03-13 18:47:52 proc_lsst.multi:201 [INFO] cancelling 17020659 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17020659 on provider escience_rtx6k 2024-03-13 18:47:53 proc_lsst.multi:146 [INFO] found job 17020660 in provider escience_rtx6k proc_lsst.multi INFO: found job 17020660 in provider escience_rtx6k 2024-03-13 18:47:53 proc_lsst.multi:201 [INFO] cancelling 17020660 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17020660 on provider escience_rtx6k 2024-03-13 18:47:53 proc_lsst.multi:146 [INFO] found job 98521 in provider local proc_lsst.multi INFO: found job 98521 in provider local 2024-03-13 18:47:53 proc_lsst.multi:201 [INFO] cancelling 98521 on provider local proc_lsst.multi INFO: cancelling 98521 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 98521 parsl.providers.local.local WARNING: Failed to kill PID: 98521 and child processes on local 2024-03-13 18:47:54 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 884.8344 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20201015_B1e_science#step1_20240313T183244Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 884.8417 seconds; current memory usage: 0.395 Gibyte, delta: 0.009 Gibyte, peak delta: 0.009 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 907.1100 seconds; current memory usage: 0.395 Gibyte, delta: 0.256 Gibyte, peak delta: 0.256 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.395 Gibyte (main), 0.395 Gibyte (largest child process) Run Id: None Run Name: DEEP_20201015_B1e_science#step1_20240313T183244Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 15m9.636s user 3m22.405s sys 2m13.694s